readme

documents/docs/creating_app_agent/demonstration_provision.md

@@ -0,0 +1,71 @@
+## Provide Human Demonstrations to the AppAgent
+
+Users or application developers can provide human demonstrations to the `AppAgent` to guide it in executing similar tasks in the future. The `AppAgent` uses these demonstrations to understand the context of the task and the steps required to execute it, effectively becoming an expert in the application.
+
+### How to Prepare Human Demonstrations for the AppAgent?
+
+Currently, UFO supports learning from user trajectories recorded by [Steps Recorder](https://support.microsoft.com/en-us/windows/record-steps-to-reproduce-a-problem-46582a9b-620f-2e36-00c9-04e25d784e47) integrated within Windows. More tools will be supported in the future.
+
+### Step 1: Recording User Demonstrations
+
+Follow the [official guidance](https://support.microsoft.com/en-us/windows/record-steps-to-reproduce-a-problem-46582a9b-620f-2e36-00c9-04e25d784e47) to use Steps Recorder to record user demonstrations.
+
+### Step 2: Add Additional Information or Comments as Needed
+
+Include any specific details or instructions for UFO to notice by adding comments. Since Steps Recorder doesn't capture typed text, include any necessary typed content in the comments as well.
+
+<p align="center">
+    <img src="../../img/add_comment.png" alt="Adding Comments in Steps Recorder"/>
+</p>
+
+
+
+### Step 3: Review and Save the Recorded Demonstrations
+
+Review the recorded steps and save them to a ZIP file. Refer to the [sample_record.zip](https://github.com/microsoft/UFO/blob/main/record_processor/example/sample_record.zip) for an example of recorded steps for a specific request, such as "sending an email to [email protected] to say hi."
+
+### Step 4: Create an Action Trajectory Indexer
+
+Once you have your demonstration record ZIP file ready, you can parse it as an example to support RAG for UFO. Follow these steps:
+
+```bash
+# Assume you are in the cloned UFO folder
+python -m record_processor -r "<your request for the demonstration>" -p "<record ZIP file path>"
+```
+
+- Replace `<your request for the demonstration>` with the specific request, such as "sending an email to [email protected] to say hi."
+- Replace `<record ZIP file path>` with the full path to the ZIP file you just created.
+
+This command will parse the record and summarize it into an execution plan. You'll see a confirmation message similar to the following:
+
+```bash
+Here are the plans summarized from your demonstration:
+Plan [1]
+(1) Input the email address '[email protected]' in the 'To' field.
+(2) Input the subject of the email. I need to input 'Greetings'.
+(3) Input the content of the email. I need to input 'Hello,\nI hope this message finds you well. I am writing to send you a warm greeting and to wish you a great day.\nBest regards.'
+(4) Click the Send button to send the email.
+Plan [2]
+(1) ***
+(2) ***
+(3) ***
+Plan [3]
+(1) ***
+(2) ***
+(3) ***
+Would you like to save any one of them as a future reference for the agent? Press [1] [2] [3] to save the corresponding plan, or press any other key to skip.
+```
+
+Press `1` to save the plan into its memory for future reference. A sample can be found [here](https://github.com/microsoft/UFO/blob/main/vectordb/demonstration/example.yaml).
+
+You can view a demonstration video below:
+
+<iframe width="560" height="315" src="https://github.com/yunhao0204/UFO/assets/59384816/0146f83e-1b5e-4933-8985-fe3f24ec4777" frameborder="0" allowfullscreen></iframe>
+
+<br>
+
+### How to Use Human Demonstrations to Enhance the AppAgent?
+
+After creating the offline indexer, refer to the [Learning from User Demonstrations](../advanced_usage/reinforce_appagent/learning_from_demonstration.md) section for guidance on how to use human demonstrations to enhance the AppAgent.
+
+---

documents/docs/creating_app_agent/help_document_provision.md

@@ -0,0 +1,42 @@
+# Providing Help Documents to the AppAgent
+
+Help documents provide guidance to the `AppAgent` in executing specific tasks. The `AppAgent` uses these documents to understand the context of the task and the steps required to execute it, effectively becoming an expert in the application.
+
+## How to Provide Help Documents to the AppAgent?
+
+### Step 1: Prepare Help Documents and Metadata
+
+Currently, UFO supports processing help documents in XML format, which is the default format for official help documents of Microsoft apps. More formats will be supported in the future.
+
+To create a dedicated document for a specific task of an app, save it in a file named, for example, `task.xml`. This document should be accompanied by a metadata file with the same prefix but with the `.meta` extension, such as `task.xml.meta`. The metadata file should include:
+
+- `title`: Describes the task at a high level.
+- `Content-Summary`: Summarizes the content of the help document.
+
+These two files are used for similarity search with user requests, so it is important to write them carefully. Examples of a help document and its metadata can be found [here](https://github.com/microsoft/UFO/blob/main/learner/doc_example/ppt-copilot.xml) and [here](https://github.com/microsoft/UFO/blob/main/learner/doc_example/ppt-copilot.xml.meta).
+
+### Step 2: Place Help Documents in the AppAgent Directory
+
+Once you have prepared all help documents and their metadata, place them into a folder. Sub-folders for the help documents are allowed, but ensure that each help document and its corresponding metadata are placed in the same directory.
+
+### Step 3: Create a Help Document Indexer
+
+After organizing your documents in a folder named `path_of_the_docs`, you can create an offline indexer to support RAG for UFO. Follow these steps:
+
+```bash
+# Assume you are in the cloned UFO folder
+python -m learner --app <app_name> --docs <path_of_the_docs>
+```
+
+- Replace `<app_name>` with the name of the application, such as PowerPoint or WeChat.
+- Replace `<path_of_the_docs>` with the full path to the folder containing all your documents.
+
+This command will create an offline indexer for all documents in the `path_of_the_docs` folder using Faiss and embedding with sentence transformer (additional embeddings will be supported soon). By default, the created index will be placed [here](https://github.com/microsoft/UFO/tree/main/vectordb/docs).
+
+!!! note
+    Ensure the `app_name` is accurately defined, as it is used to match the offline indexer in online RAG.
+
+
+### How to Use Help Documents to Enhance the AppAgent?
+
+After creating the offline indexer, you can find the guidance on how to use the help documents to enhance the AppAgent in the [Learning from Help Documents](../advanced_usage/reinforce_appagent/learning_from_help_document.md) section.