这款交互式 jupyter notebook 使用 Langchain 将虚构的工作场所文档拆分为段落 (chunks)，并使用 OpenAI 将这些段落转换为嵌入并将其存储到 Elasticsearch 中。然后，当我们提出问题时，我们从向量存储中检索相关段落，并使用 langchain 和 OpenAI 提供问题的摘要。
安装
Elasticsearch 及 Kibana
如果你还没有安装好自己的 Elasticsearch 及 Kibana，请参考如下的链接来进行安装：
-
如何在 Linux，MacOS 及 Windows 上进行安装 Elasticsearch
-
Kibana：如何在 Linux，MacOS 及 Windows 上安装 Elastic 栈中的 Kibana
在安装的时候，我们可以选择 Elastic Stack 8.x 的安装指南来进行安装。在本博文中，我将使用最新的 Elastic Stack 8.10 来进行展示。
在安装 Elasticsearch 的过程中，我们需要记下如下的信息：
Python 安装包
在本演示中，我们将使用 Python 来进行展示。我们需要安装访问 Elasticsearch 相应的安装包 elasticsearch：
python3 -m pip install -qU langchain openai elasticsearch tiktoken
我们将使用 Jupyter Notebook 来进行展示。
1. $ pwd
2. /Users/liuxg/python/elser
3. $ jupyter notebook
准备数据
我们准备如下的数据：
sample_data.json
1. [
2. {
3. "content": "Effective: March 2020nPurposennThe purpose of this full-time work-from-home policy is to provide guidelines and support for employees to conduct their work remotely, ensuring the continuity and productivity of business operations during the COVID-19 pandemic and beyond.nScopennThis policy applies to all employees who are eligible for remote work as determined by their role and responsibilities. It is designed to allow employees to work from home full time while maintaining the same level of performance and collaboration as they would in the office.nEligibilitynnEmployees who can perform their work duties remotely and have received approval from their direct supervisor and the HR department are eligible for this work-from-home arrangement.nEquipment and ResourcesnnThe necessary equipment and resources will be provided to employees for remote work, including a company-issued laptop, software licenses, and access to secure communication tools. Employees are responsible for maintaining and protecting the company's equipment and data.nWorkspacennEmployees working from home are responsible for creating a comfortable and safe workspace that is conducive to productivity. This includes ensuring that their home office is ergonomically designed, well-lit, and free from distractions.nCommunicationnnEffective communication is vital for successful remote work. Employees are expected to maintain regular communication with their supervisors, colleagues, and team members through email, phone calls, video conferences, and other approved communication tools.nWork Hours and AvailabilitynnEmployees are expected to maintain their regular work hours and be available during normal business hours, unless otherwise agreed upon with their supervisor. Any changes to work hours or availability must be communicated to the employee's supervisor and the HR department.nPerformance ExpectationsnnEmployees working from home are expected to maintain the same level of performance and productivity as if they were working in the office. Supervisors and team members will collaborate to establish clear expectations and goals for remote work.nTime Tracking and OvertimennEmployees are required to accurately track their work hours using the company's time tracking system. Non-exempt employees must obtain approval from their supervisor before working overtime.nConfidentiality and Data SecuritynnEmployees must adhere to the company's confidentiality and data security policies while working from home. This includes safeguarding sensitive information, securing personal devices and internet connections, and reporting any security breaches to the IT department.nHealth and Well-beingnnThe company encourages employees to prioritize their health and well-being while working from home. This includes taking regular breaks, maintaining a work-life balance, and seeking support from supervisors and colleagues when needed.nPolicy Review and UpdatesnnThis work-from-home policy will be reviewed periodically and updated as necessary, taking into account changes in public health guidance, business needs, and employee feedback.nQuestions and ConcernsnnEmployees are encouraged to direct any questions or concerns about this policy to their supervisor or the HR department.n",
4. "summary": "This policy outlines the guidelines for full-time remote work, including eligibility, equipment and resources, workspace requirements, communication expectations, performance expectations, time tracking and overtime, confidentiality and data security, health and well-being, and policy reviews and updates. Employees are encouraged to direct any questions or concerns",
5. "name": "Work From Home Policy",
6. "url": "./sharepoint/Work from home policy.txt",
7. "created_on": "2020-03-01",
8. "updated_at": "2020-03-01",
9. "category": "teams",
10. "_run_ml_inference": true,
11. "rolePermissions": ["demo", "manager"]
12. },
13. {
14. "content": "Starting May 2022, the company will be implementing a two-day in-office work requirement per week for all eligible employees. Please coordinate with your supervisor and HR department to schedule your in-office workdays while continuing to follow all safety protocols.n",
15. "summary": "Starting May 2022, employees will need to work two days a week in the office. Coordinate with your supervisor and HR department for these days while following safety protocols.",
16. "name": "April Work From Home Update",
17. "url": "./sharepoint/April work from home update.txt",
18. "created_on": "2022-04-29",
19. "updated_at": "2022-04-29",
20. "category": "teams",
21. "_run_ml_inference": true,
22. "rolePermissions": ["demo", "manager"]
23. },
24. {
25. "content": "As we continue to prioritize the well-being of our employees, we are making a slight adjustment to our hybrid work policy. Starting May 1, 2023, employees will be required to work from the office three days a week, with two days designated for remote work. Please communicate with your supervisor and HR department to establish your updated in-office workdays.n",
26. "summary": "Starting May 1, 2023, our hybrid work policy will require employees to work from the office three days a week and two days remotely.",
27. "name": "Wfh Policy Update May 2023",
28. "url": "./sharepoint/WFH policy update May 2023.txt",
29. "created_on": "2023-05-01",
30. "updated_at": "2023-05-01",
31. "category": "teams",
32. "_run_ml_inference": true,
33. "rolePermissions": ["demo", "manager"]
34. },
35. {
36. "content": "Executive Summary:nThis sales strategy document outlines the key objectives, focus areas, and action plans for our tech company's sales operations in fiscal year 2024. Our primary goal is to increase revenue, expand market share, and strengthen customer relationships in our target markets.nnI. Objectives for Fiscal Year 2024nnIncrease revenue by 20% compared to fiscal year 2023.nExpand market share in key segments by 15%.nRetain 95% of existing customers and increase customer satisfaction ratings.nLaunch at least two new products or services in high-demand market segments.nnII. Focus AreasnA. Target Markets:nContinue to serve existing markets with a focus on high-growth industries.nIdentify and penetrate new markets with high potential for our products and services.nnB. Customer Segmentation:nStrengthen relationships with key accounts and strategic partners.nPursue new customers in underserved market segments.nDevelop tailored offerings for different customer segments based on their needs and preferences.nnC. Product/Service Portfolio:nOptimize the existing product/service portfolio by focusing on high-demand solutions.nDevelop and launch innovative products/services in emerging technology areas.nEnhance post-sales support and customer service to improve customer satisfaction.nnIII. Action PlansnA. Sales Team Development:nExpand the sales team to cover new markets and industries.nProvide ongoing training to sales staff on product knowledge, sales techniques, and industry trends.nImplement a performance-based incentive system to reward top performers.nnB. Marketing and Promotion:nDevelop targeted marketing campaigns for different customer segments and industries.nLeverage digital marketing channels to increase brand visibility and lead generation.nParticipate in industry events and trade shows to showcase our products and services.nnC. Partner Ecosystem:nStrengthen existing partnerships and establish new strategic alliances to expand market reach.nCollaborate with partners on joint marketing and sales initiatives.nProvide partner training and support to ensure they effectively represent our products and services.nnD. Customer Success:nImplement a proactive customer success program to improve customer retention and satisfaction.nDevelop a dedicated customer support team to address customer inquiries and concerns promptly.nCollect and analyze customer feedback to identify areas for improvement in our products, services, and processes.nnIV. Monitoring and EvaluationnEstablish key performance indicators (KPIs) to track progress toward our objectives.nConduct regular sales team meetings to review performance, share best practices, and address challenges.nConduct quarterly reviews of our sales strategy to ensure alignment with market trends and adjust as needed.nnBy following this sales strategy for fiscal year 2024, our tech company aims to achieve significant growth and success in our target markets, while also providing exceptional value and service to our customers.n",
37. "summary": "This sales strategy document outlines objectives, focus areas, and action plans for our tech company's sales operations in fiscal year 2024. Our primary goal is to increase revenue, expand market share, and strengthen customer relationships in our target markets. Focus areas include targeting new markets, segmenting customers, enhancing",
38. "name": "Fy2024 Company Sales Strategy",
39. "url": "./sharepoint/FY2024 Company Sales Strategy.txt",
40. "category": "teams",
41. "created_on": "2023-04-15",
42. "updated_at": "2023-04-15",
43. "_run_ml_inference": true,
44. "rolePermissions": ["demo", "manager"]
45. },
46. {
47. "content": "PurposennThe purpose of this vacation policy is to outline the guidelines and procedures for requesting and taking time off from work for personal and leisure purposes. This policy aims to promote a healthy work-life balance and encourage employees to take time to rest and recharge.nScopennThis policy applies to all full-time and part-time employees who have completed their probationary period.nVacation AccrualnnFull-time employees accrue vacation time at a rate of [X hours] per month, equivalent to [Y days] per year. Part-time employees accrue vacation time on a pro-rata basis, calculated according to their scheduled work hours.nnVacation time will begin to accrue from the first day of employment, but employees are eligible to take vacation time only after completing their probationary period. Unused vacation time will be carried over to the next year, up to a maximum of [Z days]. Any additional unused vacation time will be forfeited.nVacation SchedulingnnEmployees are required to submit vacation requests to their supervisor at least [A weeks] in advance, specifying the start and end dates of their vacation. Supervisors will review and approve vacation requests based on business needs, ensuring adequate coverage during the employee's absence.nnEmployees are encouraged to plan their vacations around the company's peak and non-peak periods to minimize disruptions. Vacation requests during peak periods may be subject to limitations and require additional advance notice.nVacation PaynnEmployees will receive their regular pay during their approved vacation time. Vacation pay will be calculated based on the employee's average earnings over the [B weeks] preceding their vacation.nUnplanned Absences and Vacation TimennIn the event of an unplanned absence due to illness or personal emergencies, employees may use their accrued vacation time, subject to supervisor approval. Employees must inform their supervisor as soon as possible and provide any required documentation upon their return to work.nVacation Time and Termination of EmploymentnnIf an employee's employment is terminated, they will be paid out for any unused vacation time, calculated based on their current rate of pay.nPolicy Review and UpdatesnnThis vacation policy will be reviewed periodically and updated as necessary, taking into account changes in labor laws, business needs, and employee feedback.nQuestions and ConcernsnnEmployees are encouraged to direct any questions or concerns about this policy to their supervisor or the HR department.n",
48. "summary": ": This policy outlines the guidelines and procedures for requesting and taking time off from work for personal and leisure purposes. Full-time employees accrue vacation time at a rate of [X hours] per month, equivalent to [Y days] per year. Vacation requests must be submitted to supervisors at least",
49. "name": "Company Vacation Policy",
50. "url": "https://enterprisesearch.sharepoint.com/:t:/s/MSBuilddemo/ES6rw9bKZxVBobG1WUoJpikBF9Bhx1pw_GvJWbsg-Z_HNA?e=faSHVt",
51. "created_on": "2018-04-15",
52. "updated_at": "2018-04-16",
53. "category": "sharepoint",
54. "_run_ml_inference": true,
55. "rolePermissions": ["demo", "manager"]
56. },
58. {
59. "content": "This career leveling matrix provides a framework for understanding the various roles and responsibilities of Software Engineers, as well as the skills and experience required for each level. This matrix is intended to support employee development, facilitate performance evaluations, and provide a clear career progression path.nJunior Software EngineernnResponsibilities:nCollaborate with team members to design, develop, and maintain software applications and components.nWrite clean, well-structured, and efficient code following established coding standards.nParticipate in code reviews, providing and receiving constructive feedback.nTroubleshoot and resolve software defects and issues.nAssist with the creation of technical documentation.nContinuously learn and stay up-to-date with new technologies and best practices.nnSkills & Experience:nBacheloru2019s degree in Computer Science or a related field, or equivalent work experience.nBasic understanding of software development principles and methodologies.nProficiency in at least one programming language.nStrong problem-solving and analytical skills.nEffective communication and collaboration skills.nEagerness to learn and grow within the field.nSenior Software EngineernnResponsibilities:nDesign, develop, and maintain complex software applications and components.nLead and mentor junior team members in software development best practices and techniques.nConduct code reviews and ensure adherence to coding standards and best practices.nCollaborate with cross-functional teams to define, design, and deliver software solutions.nIdentify, troubleshoot, and resolve complex software defects and issues.nContribute to the creation and maintenance of technical documentation.nEvaluate and recommend new technologies, tools, and practices to improve software quality and efficiency.nnSkills & Experience:nBacheloru2019s degree in Computer Science or a related field, or equivalent work experience.n5+ years of software development experience.nProficiency in multiple programming languages and technologies.nDemonstrated ability to design and implement complex software solutions.nStrong leadership, mentoring, and collaboration skills.nExcellent problem-solving, analytical, and communication skills.nPrincipal Software EngineernnResponsibilities:nLead the design, development, and maintenance of large-scale, mission-critical software applications and components.nProvide technical leadership and mentorship to software engineering teams.nDrive the adoption of advanced software development practices and technologies.nCollaborate with product management, architecture, and other stakeholders to define and deliver strategic software initiatives.nIdentify, troubleshoot, and resolve the most complex software defects and issues.nCreate and maintain technical documentation, including architectural designs and best practice guidelines.nRepresent [Company Name] as a thought leader in the software engineering community, including speaking at conferences, publishing articles, and contributing to open-source projects.nnSkills & Experience:nBacheloru2019s degree in Computer Science or a related field, or equivalent work experience.n10+ years of software development experience, with a focus on large-scale, mission-critical applications.nExpertise in multiple programming languages, technologies, and software development methodologies.nProven ability to lead and mentor high-performing software engineering teams.nExceptional problem-solving, analytical, and communication skills.nStrong business acumen and ability to influence decision-making at the executive level.nnBy following this career leveling matrix, we aim to support the growth and development of Software Engineers, enabling them to reach their full potential and contribute meaningfully to the success of the organization.n",
60. "summary": "nThis career leveling matrix provides a framework for understanding the various roles and responsibilities of Software Engineers, as well as the skills and experience required for each level. It is intended to support employee development, facilitate performance evaluations, and provide a clear career progression path.",
61. "name": "Swe Career Matrix",
62. "url": "https://enterprisesearch.sharepoint.com/:t:/s/MSBuilddemo/EVYuEyRhHh5Aqc3a39sqbGcBkqKIHRWtJBjjUjNs6snpMg?e=nv1mf4",
63. "created_on": "2018-04-15",
64. "category": "sharepoint",
65. "_run_ml_inference": true,
66. "rolePermissions": ["demo", "manager"]
67. },
68. {
69. "content": "Title: Working with the Sales Team as an Engineer in a Tech CompanynnIntroduction:nAs an engineer in a tech company, collaboration with the sales team is essential to ensure the success of the company's products and services. This guidance document aims to provide an overview of how engineers can effectively work with the sales team, fostering a positive and productive working environment.nUnderstanding the Sales Team's Role:nThe sales team is responsible for promoting and selling the company's products and services to potential clients. Their role involves establishing relationships with customers, understanding their needs, and ensuring that the offered solutions align with their requirements.nnAs an engineer, it is important to understand the sales team's goals and objectives, as this will help you to provide them with the necessary information, tools, and support to successfully sell your company's products and services.nCommunication:nEffective communication is key to successfully working with the sales team. Make sure to maintain open lines of communication, and be responsive to their questions and concerns. This includes:nna. Attending sales meetings and conference calls when required.nb. Providing regular product updates and training sessions to the sales team.nc. Being available to answer technical questions and clarifications.nCollaboration:nCollaborate with the sales team in developing and refining sales materials, such as product presentations, demos, and technical documents. This will ensure that the sales team has accurate and up-to-date information to present to clients.nnAdditionally, work closely with the sales team on customer projects or product customizations, providing technical guidance, and ensuring that the solutions meet the customer's requirements.nCustomer Engagement:nAt times, engineers may be asked to join sales meetings or calls with potential clients to provide technical expertise. In these situations, it is important to:nna. Be prepared and understand the customer's needs and pain points.nb. Clearly explain the technical aspects of the product or solution in a simple language that the customer can understand.nc. Address any concerns or questions the customer may have.nContinuous Improvement:nActively seek feedback from the sales team regarding product performance, customer experiences, and market trends. Use this feedback to identify areas of improvement and collaborate with other engineers to enhance the product or service offerings.nMutual Respect and Support:nIt is essential to treat your colleagues in the sales team with respect and professionalism. Recognize and appreciate their efforts in promoting and selling the company's products and services. In turn, the sales team should also respect and appreciate the technical expertise and knowledge of the engineering team.nnBy working together, both the engineering and sales teams can contribute to the overall success of the company.nnConclusion:nCollaboration between engineers and the sales team is crucial for a tech company's success. By understanding each other's roles, maintaining effective communication, collaborating on projects, and supporting one another, both teams can work together to achieve the company's goals and ensure customer satisfaction.n",
70. "summary": ": This guide provides an overview of how engineers can effectively collaborate with the sales team to ensure the success of a tech company. It includes understanding the sales team's role, communicating and collaborating on projects, engaging customers, and providing mutual respect and support.",
71. "name": "Sales Engineering Collaboration",
72. "url": "https://enterprisesearch.sharepoint.com/:t:/s/MSBuilddemo/EW21-KJnfHBFoRiF49_uJMcBfHyPKimuPOFsCcJypQWaBQ?e=mGdIqe",
73. "created_on": "2019-04-15",
74. "category": "sharepoint",
75. "_run_ml_inference": true,
76. "rolePermissions": ["demo", "manager"]
77. },
78. {
79. "content": "PurposenThe purpose of this Intellectual Property Policy is to establish guidelines and procedures for the ownership, protection, and utilization of intellectual property generated by employees during their employment. This policy aims to encourage creativity and innovation while ensuring that the interests of both the company and its employees are protected.nnScopenThis policy applies to all employees, including full-time, part-time, temporary, and contract employees.nnDefinitionsna. Intellectual Property (IP): Refers to creations of the mind, such as inventions, literary and artistic works, designs, symbols, and images, that are protected by copyright, trademark, patent, or other forms of legal protection.nb. Company Time: Refers to the time during which an employee is actively engaged in performing their job duties.nc. Outside Company Time: Refers to the time during which an employee is not engaged in performing their job duties.nnOwnership of Intellectual Propertyna. Work Generated on Company Timeni. Any intellectual property created, conceived, or developed by an employee during company time or using company resources, equipment, or facilities shall be considered the property of the Company.nii. Employees are required to promptly disclose any such intellectual property to their supervisor or the appropriate department head.nb. Work Generated Outside Company Timeni. Intellectual property created, conceived, or developed by an employee outside of company time and without the use of company resources, equipment, or facilities shall generally remain the property of the employee.nii. However, if the intellectual property is directly related to the employee's job responsibilities, or if the employee has used company resources, equipment, or facilities in its creation, it may be considered the property of the Company.nProtection and Utilization of Intellectual Propertyna. The Company shall have the right to protect, license, and commercialize any intellectual property owned by the company as it deems appropriate.nb. Employees are expected to cooperate with the Company in obtaining any necessary legal protection for intellectual property owned by the company, including by signing any documents or providing any necessary information or assistance.nConfidentialitynEmployees are expected to maintain the confidentiality of any intellectual property owned by the Company and not disclose it to any third parties without the express written consent of an authorized representative of the company.nEmployee AcknowledgmentnAll employees are required to sign an acknowledgment of this Intellectual Property Policy as a condition of their employment with [Company Name]. By signing the acknowledgment, employees agree to abide by the terms of this policy and understand that any violations may result in disciplinary action, up to and including termination of employment.nPolicy ReviewnThis Intellectual Property Policy shall be reviewed periodically and may be amended as necessary to ensure its continued effectiveness and compliance with applicable laws and regulations. Employees will be notified of any significant changes to this policy.n",
80. "summary": "This Intellectual Property Policy outlines guidelines and procedures for the ownership, protection, and utilization of intellectual property generated by employees during their employment. It establishes the company's ownership of work generated on company time, while recognizing employee ownership of work generated outside of company time without the use of company resources. The policy",
81. "name": "Intellectual Property Policy",
82. "url": "https://enterprisesearch.sharepoint.com/:t:/s/MSBuilddemo/EWz3cYEVdzBNsiHsYbKhms4BVYGhravyrUw3T3lzxL4pTg?e=mPIgbO",
83. "created_on": "2021-06-15",
84. "category": "sharepoint",
85. "_run_ml_inference": true,
86. "rolePermissions": ["demo", "manager"]
87. },
89. {
90. "content": "Code of ConductnPurposennThe purpose of this code of conduct is to establish guidelines for professional and ethical behavior in the workplace. It outlines the principles and values that all employees are expected to uphold in their interactions with colleagues, customers, partners, and other stakeholders.nScopennThis code of conduct applies to all employees, contractors, and volunteers within the organization, regardless of their role or seniority.nCore ValuesnnEmployees are expected to adhere to the following core values:nna. Integrity: Act honestly, ethically, and in the best interests of the organization at all times.nb. Respect: Treat all individuals with dignity, courtesy, and fairness, regardless of their background, beliefs, or position.nc. Accountability: Take responsibility for one's actions and decisions, and be willing to learn from mistakes.nd. Collaboration: Work cooperatively with colleagues and partners to achieve shared goals and promote a positive work environment.ne. Excellence: Strive for the highest standards of performance and continuously seek opportunities for improvement.nCompliance with Laws and RegulationsnnEmployees must comply with all applicable laws, regulations, and organizational policies in the course of their work. This includes, but is not limited to, employment laws, data protection regulations, and industry-specific guidelines.nConflicts of InterestnnEmployees should avoid situations where their personal interests may conflict with or influence their professional judgment. If a potential conflict of interest arises, employees must disclose it to their supervisor or the appropriate authority within the organization.nConfidentiality and Information SecuritynnEmployees are responsible for safeguarding the organization's confidential information, as well as any sensitive information entrusted to them by clients, partners, or other third parties. This includes adhering to data protection policies and using secure communication channels.nHarassment and DiscriminationnnThe organization is committed to providing a workplace free from harassment, discrimination, and bullying. Employees are expected to treat others with respect and report any incidents of inappropriate behavior to their supervisor or the human resources department.nHealth and SafetynnEmployees must follow all health and safety guidelines and procedures to maintain a safe and healthy work environment. This includes reporting any hazards or unsafe conditions to the appropriate personnel.nUse of Company ResourcesnnEmployees are expected to use company resources, including time, equipment, and funds, responsibly and for their intended purposes. Misuse or theft of company resources is strictly prohibited.nReporting ViolationsnnEmployees have a responsibility to report any suspected violations of this code of conduct, as well as any illegal or unethical behavior, to their supervisor or the appropriate authority within the organization. The organization will protect the confidentiality of employees who report violations and will not tolerate retaliation against those who raise concerns.nConsequences of Non-CompliancennFailure to adhere to this code of conduct may result in disciplinary action, up to and including termination of employment. The organization reserves the right to take legal action against individuals who engage in illegal or unethical conduct.nPolicy Review and UpdatesnnThis code of conduct will be reviewed periodically and updated as necessary to ensure it remains relevant and effective in promoting ethical behavior and professional standards within the organization.nQuestions and ConcernsnnEmployees are encouraged to seek guidance from their supervisor or the human resources department if they have questions or concerns about this code of conduct or its application to specific situations.n",
91. "summary": "This code of conduct outlines the principles and values that all employees are expected to uphold in their interactions with colleagues, customers, partners, and other stakeholders. It sets out core values such as integrity, respect, accountability, collaboration and excellence. Employees must comply with all applicable laws, regulations, and organizational",
92. "name": "Code Of Conduct",
93. "url": "https://enterprisesearch.sharepoint.com/:t:/s/MSBuilddemo/ER3xmeKaZ_pAqPeJWyyNR0QBg6QmoWIGPhwfEyCABWHrPA?e=cvzrgV",
94. "created_on": "2018-01-12",
95. "category": "sharepoint",
96. "_run_ml_inference": true,
97. "rolePermissions": ["demo", "manager"]
98. },
99. {
100. "content": "Content:nThe purpose of this office pet policy is to outline the guidelines and procedures for bringing pets into the workplace. This policy aims to create a positive and inclusive work environment while ensuring the comfort, safety, and well-being of all employees, visitors, and pets.nScopennThis policy applies to all employees who wish to bring their pets to the office. Pets covered under this policy include dogs, cats, and other small, non-exotic animals, subject to approval by the HR department.nPet Approval ProcessnnEmployees must obtain prior approval from their supervisor and the HR department before bringing their pets to the office. The approval process includes:nna. Submitting a written request, including a description of the pet, its breed, age, and temperament.nb. Providing proof of up-to-date vaccinations and any required licenses or permits.nc. Obtaining written consent from all employees who share the workspace with the pet owner.nnThe HR department reserves the right to deny or revoke pet approval based on the specific circumstances or concerns raised by other employees.nPet Behavior and SupervisionnnEmployees are responsible for the behavior and well-being of their pets while in the office. Pets must be:nna. Well-behaved, non-aggressive, and not disruptive to the work environment.nb. House-trained and able to eliminate waste in designated areas outside the office.nc. Kept on a leash or in a secure enclosure when not in the employee's immediate work area.nnEmployees must closely supervise their pets and promptly address any issues or concerns raised by other staff members.nAllergies and PhobiasnnEmployees with allergies or phobias related to pets must inform the HR department, which will work with the affected employees and pet owners to find a suitable solution. This may include adjusting workspaces, limiting the number or types of pets allowed, or implementing additional safety measures.nCleanliness and HygienennEmployees are responsible for maintaining a clean and hygienic work environment. This includes:nna. Cleaning up after their pets, both indoors and outdoors.nb. Regularly grooming their pets to minimize shedding and odors.nc. Ensuring their pets are free of pests, such as fleas and ticks.nLiabilitynnPet owners are liable for any damage or injury caused by their pets. Employees are encouraged to obtain pet liability insurance to cover potential incidents.nRestricted AreasnnPets are not allowed in certain areas of the office, including meeting rooms, restrooms, kitchen and dining areas, and any other designated spaces. Signage will be posted to indicate these restricted areas.nPolicy Review and UpdatesnnThis office pet policy will be reviewed periodically and updated as necessary, taking into account employee feedback, changes in legislation, and best practices for maintaining a safe and inclusive work environment.nQuestions and ConcernsnnEmployees are encouraged to direct any questions or concerns about this policy to their supervisor or the HR department.n",
101. "summary": "This policy outlines the guidelines and procedures for bringing pets into the workplace. It covers approval process, pet behavior and supervision, allergies and phobias, cleanliness and hygiene, liability, restricted areas, and policy review. Employees must obtain prior approval from their supervisor and the HR department before bringing their",
102. "name": "Office Pet Policy",
103. "url": "https://enterprisesearch.sharepoint.com/:t:/s/MSBuilddemo/ETf-69wBeaZJpAn3CY7ExRABQWvav-p24VOnB6C0A4l2pQ?e=X72WuK",
104. "created_on": "2018-01-12",
105. "category": "sharepoint",
106. "_run_ml_inference": true,
107. "rolePermissions": ["demo", "manager"]
108. },
109. {
110. "content": "Performance Management PolicynPurpose and ScopenThe purpose of this Performance Management Policy is to establish a consistent and transparent process for evaluating, recognizing, and rewarding employee performance. This policy applies to all employees and aims to foster a culture of continuous improvement, professional growth, and open communication between employees and management.nPerformance Planning and Goal SettingnAt the beginning of each performance cycle, employees and their supervisors will collaborate to set clear, achievable, and measurable performance goals. These goals should align with the companyu2019s strategic objectives and take into account the employeeu2019s job responsibilities, professional development, and career aspirations.nOngoing Feedback and CommunicationnThroughout the performance cycle, employees and supervisors are encouraged to engage in regular, constructive feedback and open communication. This includes discussing progress towards goals, addressing challenges, and identifying opportunities for improvement or additional support. Regular check-ins and updates help ensure that employees stay on track and receive the guidance they need to succeed.nPerformance EvaluationnAt the end of each performance cycle, employees will participate in a formal performance evaluation with their supervisor. This evaluation will assess the employeeu2019s overall performance, including their achievements, areas for improvement, and progress towards goals. Both the employee and supervisor should come prepared to discuss specific examples, accomplishments, and challenges from the performance period.nPerformance RatingsnBased on the performance evaluation, employees will receive a performance rating that reflects their overall performance during the cycle. The rating system should be clearly defined and consistently applied across the organization. Performance ratings will be used to inform decisions regarding promotions, salary increases, and other rewards or recognition.nPromotions and AdvancementsnHigh-performing employees who consistently demonstrate strong performance, leadership, and a commitment to the companyu2019s values may be considered for promotions or other advancement opportunities. Promotions will be based on factors such as performance ratings, skills, experience, and the needs of the organization. Employees interested in pursuing a promotion should discuss their career goals and development plans with their supervisor.nPerformance Improvement PlansnEmployees who receive a low performance rating or are struggling to meet their performance goals may be placed on a Performance Improvement Plan (PIP). A PIP is a structured plan designed to help the employee address specific areas of concern, set achievable improvement goals, and receive additional support or resources as needed. Employees on a PIP will be closely monitored and re-evaluated at the end of the improvement period to determine if satisfactory progress has been made.nRecognition and RewardsnOur company believes in recognizing and rewarding employees for their hard work and dedication. In addition to promotions and salary increases, employees may be eligible for other forms of recognition or rewards based on their performance. This may include bonuses, awards, or other incentives designed to motivate and celebrate employee achievements. The specific criteria and eligibility for these rewards will be communicated by the HR department or management.n",
111. "summary": "This Performance Management Policy outlines a consistent and transparent process for evaluating, recognizing, and rewarding employees. It includes goal setting, ongoing feedback, performance evaluations, ratings, promotions, and rewards. The policy applies to all employees and encourages open communication and professional growth.",
112. "name": "Performance Management Policy",
113. "url": "https://enterprisesearch.sharepoint.com/:t:/s/MSBuilddemo/ERsxt9p1uehJqeJu4JlxkakBavbKwcldrYv_hpv3xHikAw?e=pf5R2C",
114. "created_on": "2018-01-12",
115. "category": "sharepoint",
116. "_run_ml_inference": true,
117. "rolePermissions": ["demo", "manager"]
118. },
120. {
121. "content": "Our sales organization is structured to effectively serve our customers and achieve our business objectives across multiple regions. The organization is divided into the following main regions:nnThe Americas: This region includes the United States, Canada, Mexico, as well as Central and South America. The North America South America region (NASA) has two Area Vice-Presidents: Laura Martinez is the Area Vice-President of North America, and Gary Johnson is the Area Vice-President of South America.nnEurope: Our European sales team covers the entire continent, including the United Kingdom, Germany, France, Spain, Italy, and other countries. The team is responsible for understanding the unique market dynamics and cultural nuances, enabling them to effectively target and engage with customers across the region. The Area Vice-President for Europe is Rajesh Patel.nAsia-Pacific: This region encompasses countries such as China, Japan, South Korea, India, Australia, and New Zealand. Our sales team in the Asia-Pacific region works diligently to capitalize on growth opportunities and address the diverse needs of customers in this vast and rapidly evolving market. The Area Vice-President for Asia-Pacific is Mei Li.nMiddle East & Africa: This region comprises countries across the Middle East and Africa, such as the United Arab Emirates, Saudi Arabia, South Africa, and Nigeria. Our sales team in this region is responsible for navigating the unique market challenges and identifying opportunities to expand our presence and better serve our customers. The Area Vice-President for Middle East & Africa is Jamal Abdi.nnEach regional sales team consists of dedicated account managers, sales representatives, and support staff, led by their respective Area Vice-Presidents. They are responsible for identifying and pursuing new business opportunities, nurturing existing client relationships, and ensuring customer satisfaction. The teams collaborate closely with other departments, such as marketing, product development, and customer support, to ensure we consistently deliver high-quality products and services to our clients.n",
122. "summary": "nOur sales organization is divided into four regions: The Americas, Europe, Asia-Pacific, and Middle East & Africa. Each region is led by an Area Vice-President and consists of dedicated account managers, sales representatives, and support staff. They collaborate with other departments to ensure the delivery of high",
123. "name": "Sales Organization Overview",
124. "url": "https://enterprisesearch.sharepoint.com/:t:/s/MSBuilddemo/EYsr1eqgn9hMslMJFLR-k54BBX-O3iC26bK7xNEBtYIBkg?e=xeAjiT",
125. "created_on": "2018-01-15",
126. "category": "sharepoint",
127. "_run_ml_inference": true,
128. "rolePermissions": ["demo", "manager"]
129. },
130. {
131. "content": "Introduction:nThis document outlines the compensation bands strategy for the various teams within our IT company. The goal is to establish a fair and competitive compensation structure that aligns with industry standards, rewards performance, and attracts top talent. By implementing this strategy, we aim to foster employee satisfaction and retention while ensuring the company's overall success.nnPurpose:nThe purpose of this compensation bands strategy is to:na. Define clear guidelines for salary ranges based on job levels and market benchmarks.nb. Support equitable compensation practices across different teams.nc. Encourage employee growth and performance.nd. Enable effective budgeting and resource allocation.nnJob Levels:nTo establish a comprehensive compensation structure, we have defined distinct job levels within each team. These levels reflect varying degrees of skills, experience, and responsibilities. The levels include:na. Entry-Level: Employees with limited experience or early career professionals.nb. Intermediate-Level: Employees with moderate experience and demonstrated competence.nc. Senior-Level: Experienced employees with advanced skills and leadership capabilities.nd. Leadership-Level: Managers and team leaders responsible for strategic decision-making.nnCompensation Bands:nBased on the job levels, the following compensation bands have been established:na. Entry-Level Band: This band encompasses salary ranges for employees in entry-level positions. It aims to provide competitive compensation for individuals starting their careers within the company.nnb. Intermediate-Level Band: This band covers salary ranges for employees who have gained moderate experience and expertise in their respective roles. It rewards employees for their growing skill set and contributions.nnc. Senior-Level Band: The senior-level band includes salary ranges for experienced employees who have attained advanced skills and have a proven track record of delivering results. It reflects the increased responsibilities and expectations placed upon these individuals.nnd. Leadership-Level Band: This band comprises salary ranges for managers and team leaders responsible for guiding and overseeing their respective teams. It considers their leadership abilities, strategic thinking, and the impact they have on the company's success.nnMarket Benchmarking:nTo ensure our compensation remains competitive, regular market benchmarking will be conducted. This involves analyzing industry salary trends, regional compensation data, and market demand for specific roles. The findings will inform periodic adjustments to our compensation bands to maintain alignment with the market.nnPerformance-Based Compensation:nIn addition to the defined compensation bands, we emphasize a performance-based compensation model. Performance evaluations will be conducted regularly, and employees exceeding performance expectations will be eligible for bonuses, incentives, and salary increases. This approach rewards high achievers and motivates employees to excel in their roles.nnConclusion:nBy implementing this compensation bands strategy, our IT company aims to establish fair and competitive compensation practices that align with market standards and foster employee satisfaction. Regular evaluations and market benchmarking will enable us to adapt and refine the strategy to meet the evolving needs of our organization.",
132. "summary": "This document outlines a compensation framework for IT teams. It includes job levels, compensation bands, and performance-based incentives to ensure fair and competitive wages. Regular market benchmarking will be conducted to adjust the bands according to industry trends.",
133. "name": "Compensation Framework For It Teams",
134. "url": "https://enterprisesearch.sharepoint.com/:t:/s/MSBuilddemo/EaAFec6004tAg21g4i67rfgBBRqCm1yY7AZLLQyyaMtsEQ?e=wTMb4z",
135. "created_on": "2018-01-12",
136. "category": "sharepoint",
137. "restricted": true,
138. "_run_ml_inference": true,
139. "rolePermissions": ["manager"]
140. },
141. {
142. "content": "As an employee in Canada, it's essential to understand how to update your tax elections forms to ensure accurate tax deductions from your pay. This guide will help you navigate the process of updating your TD1 Personal Tax Credits Return form.nnStep 1: Access the TD1 formnThe TD1 form is available on the Canada Revenue Agency (CRA) website. Your employer might provide you with a paper copy or a link to the online form. You can access the form directly through the following link: https://www.canada.ca/en/revenue-agency/services/forms-publications/td1-personal-tax-credits-returns.htmlnnStep 2: Choose the correct form versionnYou'll need to fill out the federal TD1 form and, if applicable, the provincial or territorial TD1 form. Select the appropriate version based on your province or territory of residence.nnStep 3: Download and open the formnFor the best experience, download and open the TD1 form in Adobe Reader. If you have visual impairments, consider using the large print version available on the CRA website.nnStep 4: Complete the formnFill out the form by entering your personal information, such as your name, Social Insurance Number (SIN), and address. Then, go through each section to claim any personal tax credits that apply to you. These credits may include:nBasic personal amountnAmount for an eligible dependantnAmount for infirm dependants age 18 or oldernCaregiver amountnDisability amountnTuition and education amountsnnRead the instructions carefully for each section to ensure you claim the correct amounts.nnStep 5: Sign and date the formnOnce you've completed the form, sign and date it at the bottom.nnStep 6: Submit the form to your employernSubmit the completed and signed TD1 form to your employer. You can either scan and send it electronically, or provide a printed copy. Your employer will use the information on your TD1 form to calculate the correct amount of tax to be deducted from your pay.nnStep 7: Update your TD1 form as needednIt's essential to update your TD1 form whenever your personal circumstances change, such as getting married, having a child, or becoming eligible for a new tax credit. Inform your employer of these changes and submit an updated TD1 form to ensure accurate tax deductions.nnUpdating your tax elections forms is a crucial step in ensuring the correct tax deductions from your pay as a new employee in Canada. Follow this guide and keep your TD1 form up to date to avoid any discrepancies in your tax filings.n",
143. "summary": ": This guide gives a step-by-step explanation of how to update your TD1 Personal Tax Credits Return form. Access the form from the CRA website and choose the correct version based on your province or territory of residence. Download and open the form in Adobe Reader, fill out the form by entering",
144. "name": "Updating Your Tax Elections Forms",
145. "url": "./github/Updating Your Tax Elections Forms.txt",
146. "created_on": "2022-12-20",
147. "category": "github",
148. "_run_ml_inference": true,
149. "rolePermissions": ["demo", "manager"]
150. },
151. {
152. "content": "Welcome to our team! We are excited to have you on board and look forward to your valuable contributions. This onboarding guide is designed to help you get started by providing essential information about our policies, procedures, and resources. Please read through this guide carefully and reach out to the HR department if you have any questions.nIntroduction to Our Company Culture and ValuesnOur company is committed to creating a diverse, inclusive, and supportive work environment. We believe that our employees are our most valuable asset and strive to foster a culture of collaboration, innovation, and continuous learning. Our core values include:nIntegrity: We act ethically and honestly in all our interactions.nTeamwork: We work together to achieve common goals and support each other's growth.nExcellence: We strive for the highest quality in our products, services, and relationships.nInnovation: We encourage creativity and embrace change to stay ahead in the market.nRespect: We treat each other with dignity and value the unique perspectives of all our colleagues.nKey Onboarding StepsnTo ensure a smooth onboarding process, please complete the following steps within your first week:nAttend orientation: You will be invited to an orientation session to meet your colleagues and learn more about our company's history, mission, and values.nReview policies and procedures: Familiarize yourself with our employee handbook, which contains important information about our policies and procedures. Please read it thoroughly and adhere to the guidelines.nComplete required training: You may be required to complete mandatory training sessions, such as safety training or anti-harassment training. Ensure that you attend and complete these sessions as soon as possible.nUpdating Tax Elections and DocumentsnIt is crucial to ensure your tax information is accurate and up-to-date, regardless of the country you work in. Please follow these steps to update your tax elections and documents:nComplete tax forms: Fill out the necessary tax forms for your country or region, which determine the amount of income tax withheld from your paycheck. You should complete new tax forms if your personal or financial situation changes, such as marriage, divorce, or a change in the number of dependents.nSubmit regional tax forms: Depending on your location, you may be required to complete additional regional or local tax forms. Check with the HR department to determine which forms are necessary.nUpdate your address: If you move, make sure to update your address with the HR department to ensure accurate tax reporting.nBenefits EnrollmentnAs a new employee, you are eligible for various benefits, including health insurance, retirement plans, and paid time off. You will receive detailed information about our benefits package during orientation. To enroll in the benefits, please follow these steps:nReview benefits options: Carefully review the benefits package and choose the options that best meet your needs.nComplete enrollment forms: Fill out the necessary forms to enroll in your chosen benefits. Submit these forms to the HR department within 30 days of your start date.nDesignate beneficiaries: If applicable, designate beneficiaries for your life insurance and retirement plans.nGetting Settled in Your WorkspacenTo help you feel comfortable and productive in your new workspace, take the following steps:nSet up your workstation: Organize your desk, chair, and computer according to your preferences. If you require any additional equipment or accommodations, please contact the HR department.nObtain necessary supplies: Request any necessary office supplies, such as pens, notepads, or folders, from the designated supply area or by contacting the appropriate department.nFamiliarize yourself with office resources: Locate common areas, such as break rooms, restrooms, and meeting rooms. Familiarize yourself with office equipment, including printers, scanners, and telephones.n",
153. "summary": "nThis onboarding guide provides essential information to new employees on our company culture and values, key onboarding steps, tax elections and documents, benefits enrollment, and setting up their workspace.",
154. "name": "New Employee Onboarding Guide",
155. "url": "./github/New Employee Onboarding guide.txt",
156. "created_on": "2018-01-12",
157. "category": "github",
158. "_run_ml_inference": true,
159. "rolePermissions": ["demo", "manager"]
160. }
161. ]
1. $ pwd
2. /Users/liuxg/python/elser
3. $ ls
4. Multilingual semantic search.ipynb
5. NLP text search using hugging face transformer model.ipynb
6. Question Answering with Langchain and OpenAI.ipynb
7. Semantic search - ELSER.ipynb
8. Semantic search quick start.ipynb
9. books.json
10. data.json
11. sample_data.json
如上所示，我们在根目录下创建一个叫做 sample_data.json 的文件。
创建应用并展示
安装包并导入模块
1. # import modules
2. from getpass import getpass
3. from langchain.vectorstores import ElasticsearchStore
4. from langchain.embeddings.openai import OpenAIEmbeddings
5. from urllib.request import urlopen
6. from langchain.text_splitter import CharacterTextSplitter
7. from langchain.llms import OpenAI
8. from langchain.chains import RetrievalQA
9. from langchain.vectorstores import ElasticVectorSearch
10. import json
连接到 Elasticsearch
现在我们可以实例化 Elasticsearch python 客户端。我们必须提供相应的账号及证书信息：
1. from elasticsearch import Elasticsearch
3. ELASTCSEARCH_CERT_PATH = "/Users/liuxg/elastic/elasticsearch-8.10.0/config/certs/http_ca.crt"
5. OPENAI_API_KEY = getpass("OpenAI API key")
7. embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY)
9. vector_store = ElasticVectorSearch(
10. elasticsearch_url="http://localhost:9200",
11. index_,
12. embedding=embeddings,
13. )
写入数据
让我装载 sample_data.json 数据并饭系列化文档：
1. from elasticsearch import helpers
3. # Load data into a JSON object
4. with open('sample_data.json') as f:
5. workplace_docs = json.load(f)
7. print(workplace_docs)
将文档拆分为段落
我们将把文档分成段落，以提高检索的特异性，并确保我们可以在最终问答提示的上下文窗口中提供多个段落。
在这里，我们将文档分块为 800 个标记段落，其中有 400 个标记重叠。
这里我们使用一个简单的拆分器，但 Langchain 提供了更高级的拆分器来减少上下文丢失的机会。
1. metadata = []
2. content = []
4. for doc in workplace_docs:
5. content.append(doc["content"])
6. metadata.append({
7. "name": doc["name"],
8. "summary": doc["summary"]
9. })
11. text_splitter = CharacterTextSplitter(chunk_size=800, chunk_overlap=400)
12. docs = text_splitter.create_documents(content, metadatas=metadata)
将数据索引到elasticsearch中
现在我们已将每个文档拆分为大小为 800 的块，现在我们将使用 ElasticsearchStore.from_documents 将数据索引到 elasticsearch。
我们先把 Elasticsearch 集群的证书拷贝到当前目录：
1. $ pwd
2. /Users/liuxg/python/elser
3. $ cp ~/elastic/elasticsearch-8.10.0/config/certs/http_ca.crt .
4. $ ls
5. Multilingual semantic search.ipynb
6. NLP text search using hugging face transformer model.ipynb
7. Question Answering with Langchain and OpenAI.ipynb
8. Semantic search - ELSER.ipynb
9. Semantic search quick start.ipynb
10. books.json
11. data.json
12. http_ca.crt
13. sample_data.json
1. username = "elastic"
2. password = "vXDWYtL*my3vnKY9zCfL"
3. localhost = "localhost"
4. ssl_verify = {
5. "verify_certs": True,
6. "basic_auth": (username, password),
7. "ca_certs": "./http_ca.crt"
8. }
10. url = f"https://{username}:{password}@{endpoint}:9200"
13. documents = ElasticVectorSearch.from_documents( docs,
14. embedding = embeddings,
15. elasticsearch_url = url,
16. index_name = "workplace_index",
17. ssl_verify = ssl_verify)
在上面，我们需要根据自己的 Elasticsearch 安装，修改上面的账号信息以及终端访问地址.
提出问题
现在我们已经将段落存储在 Elasticsearch 中，我们现在可以提出问题来获取相关段落。
1. retriever = vector_store.as_retriever()
3. llm = OpenAI(openai_api_key=OPENAI_API_KEY)
5. qa = RetrievalQA.from_chain_type(
6. llm=llm,
7. chain_type="stuff",
8. retriever=retriever,
9. return_source_documents=True
10. )
12. ans = qa({"query": "what is the nasa sales team?"})
14. print("---- Answer ----")
15. print(ans["result"])
16. print("---- Sources ----")
17. for doc in ans["source_documents"]:
18. print(doc.metadata["name"])
19. print(doc.page_content)
1. ---- Answer ----
2. The NASA sales team is the North America South America regional sales team, led by the Area Vice-Presidents Laura Martinez and Gary Johnson.
3. ---- Sources ----
4. Sales Organization Overview
5. Our sales organization is structured to effectively serve our customers and achieve our business objectives across multiple regions. The organization is divided into the following main regions:
7. The Americas: This region includes the United States, Canada, Mexico, as well as Central and South America. The North America South America region (NASA) has two Area Vice-Presidents: Laura Martinez is the Area Vice-President of North America, and Gary Johnson is the Area Vice-President of South America.
8. Sales Organization Overview
9. Our sales organization is structured to effectively serve our customers and achieve our business objectives across multiple regions. The organization is divided into the following main regions:
11. The Americas: This region includes the United States, Canada, Mexico, as well as Central and South America. The North America South America region (NASA) has two Area Vice-Presidents: Laura Martinez is the Area Vice-President of North America, and Gary Johnson is the Area Vice-President of South America.
12. Sales Organization Overview
13. Each regional sales team consists of dedicated account managers, sales representatives, and support staff, led by their respective Area Vice-Presidents. They are responsible for identifying and pursuing new business opportunities, nurturing existing client relationships, and ensuring customer satisfaction. The teams collaborate closely with other departments, such as marketing, product development, and customer support, to ensure we consistently deliver high-quality products and services to our clients.
14. Sales Organization Overview
15. Each regional sales team consists of dedicated account managers, sales representatives, and support staff, led by their respective Area Vice-Presidents. They are responsible for identifying and pursuing new business opportunities, nurturing existing client relationships, and ensuring customer satisfaction. The teams collaborate closely with other departments, such as marketing, product development, and customer support, to ensure we consistently deliver high-quality products and services to our clients.
