初始化
This commit is contained in:
雅诺狐
674
src/chat/knowledge/LICENSE
Normal file
674
src/chat/knowledge/LICENSE
Normal file
@@ -0,0 +1,674 @@
|
||||
GNU GENERAL PUBLIC LICENSE
|
||||
Version 3, 29 June 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU General Public License is a free, copyleft license for
|
||||
software and other kinds of works.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
the GNU General Public License is intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users. We, the Free Software Foundation, use the
|
||||
GNU General Public License for most of our software; it applies also to
|
||||
any other work released this way by its authors. You can apply it to
|
||||
your programs, too.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
To protect your rights, we need to prevent others from denying you
|
||||
these rights or asking you to surrender the rights. Therefore, you have
|
||||
certain responsibilities if you distribute copies of the software, or if
|
||||
you modify it: responsibilities to respect the freedom of others.
|
||||
|
||||
For example, if you distribute copies of such a program, whether
|
||||
gratis or for a fee, you must pass on to the recipients the same
|
||||
freedoms that you received. You must make sure that they, too, receive
|
||||
or can get the source code. And you must show them these terms so they
|
||||
know their rights.
|
||||
|
||||
Developers that use the GNU GPL protect your rights with two steps:
|
||||
(1) assert copyright on the software, and (2) offer you this License
|
||||
giving you legal permission to copy, distribute and/or modify it.
|
||||
|
||||
For the developers' and authors' protection, the GPL clearly explains
|
||||
that there is no warranty for this free software. For both users' and
|
||||
authors' sake, the GPL requires that modified versions be marked as
|
||||
changed, so that their problems will not be attributed erroneously to
|
||||
authors of previous versions.
|
||||
|
||||
Some devices are designed to deny users access to install or run
|
||||
modified versions of the software inside them, although the manufacturer
|
||||
can do so. This is fundamentally incompatible with the aim of
|
||||
protecting users' freedom to change the software. The systematic
|
||||
pattern of such abuse occurs in the area of products for individuals to
|
||||
use, which is precisely where it is most unacceptable. Therefore, we
|
||||
have designed this version of the GPL to prohibit the practice for those
|
||||
products. If such problems arise substantially in other domains, we
|
||||
stand ready to extend this provision to those domains in future versions
|
||||
of the GPL, as needed to protect the freedom of users.
|
||||
|
||||
Finally, every program is threatened constantly by software patents.
|
||||
States should not allow patents to restrict development and use of
|
||||
software on general-purpose computers, but in those that do, we wish to
|
||||
avoid the special danger that patents applied to a free program could
|
||||
make it effectively proprietary. To prevent this, the GPL assures that
|
||||
patents cannot be used to render the program non-free.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Use with the GNU Affero General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU Affero General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the special requirements of the GNU Affero General Public License,
|
||||
section 13, concerning interaction through a network will apply to the
|
||||
combination as such.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU General Public License from time to time. Such new versions will
|
||||
be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If the program does terminal interaction, make it output a short
|
||||
notice like this when it starts in an interactive mode:
|
||||
|
||||
<program> Copyright (C) <year> <name of author>
|
||||
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
||||
This is free software, and you are welcome to redistribute it
|
||||
under certain conditions; type `show c' for details.
|
||||
|
||||
The hypothetical commands `show w' and `show c' should show the appropriate
|
||||
parts of the General Public License. Of course, your program's commands
|
||||
might be different; for a GUI interface, you would use an "about box".
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU GPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
|
||||
The GNU General Public License does not permit incorporating your program
|
||||
into proprietary programs. If your program is a subroutine library, you
|
||||
may consider it more useful to permit linking proprietary applications with
|
||||
the library. If this is what you want to do, use the GNU Lesser General
|
||||
Public License instead of this License. But first, please read
|
||||
<https://www.gnu.org/licenses/why-not-lgpl.html>.
|
||||
0
src/chat/knowledge/__init__.py
Normal file
0
src/chat/knowledge/__init__.py
Normal file
592
src/chat/knowledge/embedding_store.py
Normal file
592
src/chat/knowledge/embedding_store.py
Normal file
@@ -0,0 +1,592 @@
|
||||
from dataclasses import dataclass
|
||||
import json
|
||||
import os
|
||||
import math
|
||||
import asyncio
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from typing import Dict, List, Tuple
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
# import tqdm
|
||||
import faiss
|
||||
|
||||
from .utils.hash import get_sha256
|
||||
from .global_logger import logger
|
||||
from rich.traceback import install
|
||||
from rich.progress import (
|
||||
Progress,
|
||||
BarColumn,
|
||||
TimeElapsedColumn,
|
||||
TimeRemainingColumn,
|
||||
TaskProgressColumn,
|
||||
MofNCompleteColumn,
|
||||
SpinnerColumn,
|
||||
TextColumn,
|
||||
)
|
||||
from src.chat.utils.utils import get_embedding
|
||||
from src.config.config import global_config
|
||||
|
||||
|
||||
install(extra_lines=3)
|
||||
|
||||
# 多线程embedding配置常量
|
||||
DEFAULT_MAX_WORKERS = 10 # 默认最大线程数
|
||||
DEFAULT_CHUNK_SIZE = 10 # 默认每个线程处理的数据块大小
|
||||
MIN_CHUNK_SIZE = 1 # 最小分块大小
|
||||
MAX_CHUNK_SIZE = 50 # 最大分块大小
|
||||
MIN_WORKERS = 1 # 最小线程数
|
||||
MAX_WORKERS = 20 # 最大线程数
|
||||
|
||||
ROOT_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", ".."))
|
||||
EMBEDDING_DATA_DIR = os.path.join(ROOT_PATH, "data", "embedding")
|
||||
EMBEDDING_DATA_DIR_STR = str(EMBEDDING_DATA_DIR).replace("\\", "/")
|
||||
TOTAL_EMBEDDING_TIMES = 3 # 统计嵌入次数
|
||||
|
||||
# 嵌入模型测试字符串，测试模型一致性，来自开发群的聊天记录
|
||||
# 这些字符串的嵌入结果应该是固定的，不能随时间变化
|
||||
EMBEDDING_TEST_STRINGS = [
|
||||
"阿卡伊真的太好玩了，神秘性感大女同等着你",
|
||||
"你怎么知道我arc12.64了",
|
||||
"我是蕾缪乐小姐的狗",
|
||||
"关注Oct谢谢喵",
|
||||
"不是w6我不草",
|
||||
"关注千石可乐谢谢喵",
|
||||
"来玩CLANNAD，AIR，樱之诗，樱之刻谢谢喵",
|
||||
"关注墨梓柒谢谢喵",
|
||||
"Ciallo~",
|
||||
"来玩巧克甜恋谢谢喵",
|
||||
"水印",
|
||||
"我也在纠结晚饭，铁锅炒鸡听着就香！",
|
||||
"test你妈喵",
|
||||
]
|
||||
EMBEDDING_TEST_FILE = os.path.join(ROOT_PATH, "data", "embedding_model_test.json")
|
||||
EMBEDDING_SIM_THRESHOLD = 0.99
|
||||
|
||||
|
||||
def cosine_similarity(a, b):
|
||||
# 计算余弦相似度
|
||||
dot = sum(x * y for x, y in zip(a, b, strict=False))
|
||||
norm_a = math.sqrt(sum(x * x for x in a))
|
||||
norm_b = math.sqrt(sum(x * x for x in b))
|
||||
if norm_a == 0 or norm_b == 0:
|
||||
return 0.0
|
||||
return dot / (norm_a * norm_b)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EmbeddingStoreItem:
|
||||
"""嵌入库中的项"""
|
||||
|
||||
def __init__(self, item_hash: str, embedding: List[float], content: str):
|
||||
self.hash = item_hash
|
||||
self.embedding = embedding
|
||||
self.str = content
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
"""转为dict"""
|
||||
return {
|
||||
"hash": self.hash,
|
||||
"embedding": self.embedding,
|
||||
"str": self.str,
|
||||
}
|
||||
|
||||
|
||||
class EmbeddingStore:
|
||||
def __init__(self, namespace: str, dir_path: str, max_workers: int = DEFAULT_MAX_WORKERS, chunk_size: int = DEFAULT_CHUNK_SIZE):
|
||||
self.namespace = namespace
|
||||
self.dir = dir_path
|
||||
self.embedding_file_path = f"{dir_path}/{namespace}.parquet"
|
||||
self.index_file_path = f"{dir_path}/{namespace}.index"
|
||||
self.idx2hash_file_path = dir_path + "/" + namespace + "_i2h.json"
|
||||
|
||||
# 多线程配置参数验证和设置
|
||||
self.max_workers = max(MIN_WORKERS, min(MAX_WORKERS, max_workers))
|
||||
self.chunk_size = max(MIN_CHUNK_SIZE, min(MAX_CHUNK_SIZE, chunk_size))
|
||||
|
||||
# 如果配置值被调整，记录日志
|
||||
if self.max_workers != max_workers:
|
||||
logger.warning(f"max_workers 已从 {max_workers} 调整为 {self.max_workers} (范围: {MIN_WORKERS}-{MAX_WORKERS})")
|
||||
if self.chunk_size != chunk_size:
|
||||
logger.warning(f"chunk_size 已从 {chunk_size} 调整为 {self.chunk_size} (范围: {MIN_CHUNK_SIZE}-{MAX_CHUNK_SIZE})")
|
||||
|
||||
self.store = {}
|
||||
|
||||
self.faiss_index = None
|
||||
self.idx2hash = None
|
||||
|
||||
def _get_embedding(self, s: str) -> List[float]:
|
||||
"""获取字符串的嵌入向量，处理异步调用"""
|
||||
try:
|
||||
# 尝试获取当前事件循环
|
||||
asyncio.get_running_loop()
|
||||
# 如果在事件循环中，使用线程池执行
|
||||
import concurrent.futures
|
||||
|
||||
def run_in_thread():
|
||||
return asyncio.run(get_embedding(s))
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor() as executor:
|
||||
future = executor.submit(run_in_thread)
|
||||
result = future.result()
|
||||
if result is None:
|
||||
logger.error(f"获取嵌入失败: {s}")
|
||||
return []
|
||||
return result
|
||||
except RuntimeError:
|
||||
# 没有运行的事件循环，直接运行
|
||||
result = asyncio.run(get_embedding(s))
|
||||
if result is None:
|
||||
logger.error(f"获取嵌入失败: {s}")
|
||||
return []
|
||||
return result
|
||||
|
||||
def _get_embeddings_batch_threaded(self, strs: List[str], chunk_size: int = 10, max_workers: int = 10, progress_callback=None) -> List[Tuple[str, List[float]]]:
|
||||
"""使用多线程批量获取嵌入向量
|
||||
|
||||
Args:
|
||||
strs: 要获取嵌入的字符串列表
|
||||
chunk_size: 每个线程处理的数据块大小
|
||||
max_workers: 最大线程数
|
||||
progress_callback: 进度回调函数，接收一个参数表示完成的数量
|
||||
|
||||
Returns:
|
||||
包含(原始字符串, 嵌入向量)的元组列表，保持与输入顺序一致
|
||||
"""
|
||||
if not strs:
|
||||
return []
|
||||
|
||||
# 分块
|
||||
chunks = []
|
||||
for i in range(0, len(strs), chunk_size):
|
||||
chunk = strs[i:i + chunk_size]
|
||||
chunks.append((i, chunk)) # 保存起始索引以维持顺序
|
||||
|
||||
# 结果存储，使用字典按索引存储以保证顺序
|
||||
results = {}
|
||||
|
||||
def process_chunk(chunk_data):
|
||||
"""处理单个数据块的函数"""
|
||||
start_idx, chunk_strs = chunk_data
|
||||
chunk_results = []
|
||||
|
||||
# 为每个线程创建独立的LLMRequest实例
|
||||
from src.llm_models.utils_model import LLMRequest
|
||||
from src.config.config import model_config
|
||||
|
||||
try:
|
||||
# 创建线程专用的LLM实例
|
||||
llm = LLMRequest(model_set=model_config.model_task_config.embedding, request_type="embedding")
|
||||
|
||||
for i, s in enumerate(chunk_strs):
|
||||
try:
|
||||
# 直接使用异步函数
|
||||
embedding = asyncio.run(llm.get_embedding(s))
|
||||
if embedding and len(embedding) > 0:
|
||||
chunk_results.append((start_idx + i, s, embedding[0])) # embedding[0] 是实际的向量
|
||||
else:
|
||||
logger.error(f"获取嵌入失败: {s}")
|
||||
chunk_results.append((start_idx + i, s, []))
|
||||
|
||||
# 每完成一个嵌入立即更新进度
|
||||
if progress_callback:
|
||||
progress_callback(1)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取嵌入时发生异常: {s}, 错误: {e}")
|
||||
chunk_results.append((start_idx + i, s, []))
|
||||
|
||||
# 即使失败也要更新进度
|
||||
if progress_callback:
|
||||
progress_callback(1)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"创建LLM实例失败: {e}")
|
||||
# 如果创建LLM实例失败，返回空结果
|
||||
for i, s in enumerate(chunk_strs):
|
||||
chunk_results.append((start_idx + i, s, []))
|
||||
# 即使失败也要更新进度
|
||||
if progress_callback:
|
||||
progress_callback(1)
|
||||
|
||||
return chunk_results
|
||||
|
||||
# 使用线程池处理
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
# 提交所有任务
|
||||
future_to_chunk = {executor.submit(process_chunk, chunk): chunk for chunk in chunks}
|
||||
|
||||
# 收集结果（进度已在process_chunk中实时更新）
|
||||
for future in as_completed(future_to_chunk):
|
||||
try:
|
||||
chunk_results = future.result()
|
||||
for idx, s, embedding in chunk_results:
|
||||
results[idx] = (s, embedding)
|
||||
except Exception as e:
|
||||
chunk = future_to_chunk[future]
|
||||
logger.error(f"处理数据块时发生异常: {chunk}, 错误: {e}")
|
||||
# 为失败的块添加空结果
|
||||
start_idx, chunk_strs = chunk
|
||||
for i, s in enumerate(chunk_strs):
|
||||
results[start_idx + i] = (s, [])
|
||||
|
||||
# 按原始顺序返回结果
|
||||
ordered_results = []
|
||||
for i in range(len(strs)):
|
||||
if i in results:
|
||||
ordered_results.append(results[i])
|
||||
else:
|
||||
# 防止遗漏
|
||||
ordered_results.append((strs[i], []))
|
||||
|
||||
return ordered_results
|
||||
|
||||
def get_test_file_path(self):
|
||||
return EMBEDDING_TEST_FILE
|
||||
|
||||
def save_embedding_test_vectors(self):
|
||||
"""保存测试字符串的嵌入到本地（使用多线程优化）"""
|
||||
logger.info("开始保存测试字符串的嵌入向量...")
|
||||
|
||||
# 使用多线程批量获取测试字符串的嵌入
|
||||
embedding_results = self._get_embeddings_batch_threaded(
|
||||
EMBEDDING_TEST_STRINGS,
|
||||
chunk_size=min(self.chunk_size, len(EMBEDDING_TEST_STRINGS)),
|
||||
max_workers=min(self.max_workers, len(EMBEDDING_TEST_STRINGS))
|
||||
)
|
||||
|
||||
# 构建测试向量字典
|
||||
test_vectors = {}
|
||||
for idx, (s, embedding) in enumerate(embedding_results):
|
||||
if embedding:
|
||||
test_vectors[str(idx)] = embedding
|
||||
else:
|
||||
logger.error(f"获取测试字符串嵌入失败: {s}")
|
||||
# 使用原始单线程方法作为后备
|
||||
test_vectors[str(idx)] = self._get_embedding(s)
|
||||
|
||||
with open(self.get_test_file_path(), "w", encoding="utf-8") as f:
|
||||
json.dump(test_vectors, f, ensure_ascii=False, indent=2)
|
||||
|
||||
logger.info("测试字符串嵌入向量保存完成")
|
||||
|
||||
def load_embedding_test_vectors(self):
|
||||
"""加载本地保存的测试字符串嵌入"""
|
||||
path = self.get_test_file_path()
|
||||
if not os.path.exists(path):
|
||||
return None
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
def check_embedding_model_consistency(self):
|
||||
"""校验当前模型与本地嵌入模型是否一致（使用多线程优化）"""
|
||||
local_vectors = self.load_embedding_test_vectors()
|
||||
if local_vectors is None:
|
||||
logger.warning("未检测到本地嵌入模型测试文件，将保存当前模型的测试嵌入。")
|
||||
self.save_embedding_test_vectors()
|
||||
return True
|
||||
|
||||
# 检查本地向量完整性
|
||||
for idx in range(len(EMBEDDING_TEST_STRINGS)):
|
||||
if local_vectors.get(str(idx)) is None:
|
||||
logger.warning("本地嵌入模型测试文件缺失部分测试字符串，将重新保存。")
|
||||
self.save_embedding_test_vectors()
|
||||
return True
|
||||
|
||||
logger.info("开始检验嵌入模型一致性...")
|
||||
|
||||
# 使用多线程批量获取当前模型的嵌入
|
||||
embedding_results = self._get_embeddings_batch_threaded(
|
||||
EMBEDDING_TEST_STRINGS,
|
||||
chunk_size=min(self.chunk_size, len(EMBEDDING_TEST_STRINGS)),
|
||||
max_workers=min(self.max_workers, len(EMBEDDING_TEST_STRINGS))
|
||||
)
|
||||
|
||||
# 检查一致性
|
||||
for idx, (s, new_emb) in enumerate(embedding_results):
|
||||
local_emb = local_vectors.get(str(idx))
|
||||
if not new_emb:
|
||||
logger.error(f"获取测试字符串嵌入失败: {s}")
|
||||
return False
|
||||
|
||||
sim = cosine_similarity(local_emb, new_emb)
|
||||
if sim < EMBEDDING_SIM_THRESHOLD:
|
||||
logger.error(f"嵌入模型一致性校验失败，字符串: {s}, 相似度: {sim:.4f}")
|
||||
return False
|
||||
|
||||
logger.info("嵌入模型一致性校验通过。")
|
||||
return True
|
||||
|
||||
def batch_insert_strs(self, strs: List[str], times: int) -> None:
|
||||
"""向库中存入字符串（使用多线程优化）"""
|
||||
if not strs:
|
||||
return
|
||||
|
||||
total = len(strs)
|
||||
|
||||
# 过滤已存在的字符串
|
||||
new_strs = []
|
||||
for s in strs:
|
||||
item_hash = self.namespace + "-" + get_sha256(s)
|
||||
if item_hash not in self.store:
|
||||
new_strs.append(s)
|
||||
|
||||
if not new_strs:
|
||||
logger.info(f"所有字符串已存在于{self.namespace}嵌入库中，跳过处理")
|
||||
return
|
||||
|
||||
logger.info(f"需要处理 {len(new_strs)}/{total} 个新字符串")
|
||||
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TaskProgressColumn(),
|
||||
MofNCompleteColumn(),
|
||||
"•",
|
||||
TimeElapsedColumn(),
|
||||
"<",
|
||||
TimeRemainingColumn(),
|
||||
transient=False,
|
||||
) as progress:
|
||||
task = progress.add_task(f"存入嵌入库：({times}/{TOTAL_EMBEDDING_TIMES})", total=total)
|
||||
|
||||
# 首先更新已存在项的进度
|
||||
already_processed = total - len(new_strs)
|
||||
if already_processed > 0:
|
||||
progress.update(task, advance=already_processed)
|
||||
|
||||
if new_strs:
|
||||
# 使用实例配置的参数，智能调整分块和线程数
|
||||
optimal_chunk_size = max(MIN_CHUNK_SIZE, min(self.chunk_size, len(new_strs) // self.max_workers if self.max_workers > 0 else self.chunk_size))
|
||||
optimal_max_workers = min(self.max_workers, max(MIN_WORKERS, len(new_strs) // optimal_chunk_size if optimal_chunk_size > 0 else 1))
|
||||
|
||||
logger.debug(f"使用多线程处理: chunk_size={optimal_chunk_size}, max_workers={optimal_max_workers}")
|
||||
|
||||
# 定义进度更新回调函数
|
||||
def update_progress(count):
|
||||
progress.update(task, advance=count)
|
||||
|
||||
# 批量获取嵌入，并实时更新进度
|
||||
embedding_results = self._get_embeddings_batch_threaded(
|
||||
new_strs,
|
||||
chunk_size=optimal_chunk_size,
|
||||
max_workers=optimal_max_workers,
|
||||
progress_callback=update_progress
|
||||
)
|
||||
|
||||
# 存入结果（不再需要在这里更新进度，因为已经在回调中更新了）
|
||||
for s, embedding in embedding_results:
|
||||
item_hash = self.namespace + "-" + get_sha256(s)
|
||||
if embedding: # 只有成功获取到嵌入才存入
|
||||
self.store[item_hash] = EmbeddingStoreItem(item_hash, embedding, s)
|
||||
else:
|
||||
logger.warning(f"跳过存储失败的嵌入: {s[:50]}...")
|
||||
|
||||
def save_to_file(self) -> None:
|
||||
"""保存到文件"""
|
||||
data = []
|
||||
logger.info(f"正在保存{self.namespace}嵌入库到文件{self.embedding_file_path}")
|
||||
for item in self.store.values():
|
||||
data.append(item.to_dict())
|
||||
data_frame = pd.DataFrame(data)
|
||||
|
||||
if not os.path.exists(self.dir):
|
||||
os.makedirs(self.dir, exist_ok=True)
|
||||
if not os.path.exists(self.embedding_file_path):
|
||||
open(self.embedding_file_path, "w").close()
|
||||
|
||||
data_frame.to_parquet(self.embedding_file_path, engine="pyarrow", index=False)
|
||||
logger.info(f"{self.namespace}嵌入库保存成功")
|
||||
|
||||
if self.faiss_index is not None and self.idx2hash is not None:
|
||||
logger.info(f"正在保存{self.namespace}嵌入库的FaissIndex到文件{self.index_file_path}")
|
||||
faiss.write_index(self.faiss_index, self.index_file_path)
|
||||
logger.info(f"{self.namespace}嵌入库的FaissIndex保存成功")
|
||||
logger.info(f"正在保存{self.namespace}嵌入库的idx2hash映射到文件{self.idx2hash_file_path}")
|
||||
with open(self.idx2hash_file_path, "w", encoding="utf-8") as f:
|
||||
f.write(json.dumps(self.idx2hash, ensure_ascii=False, indent=4))
|
||||
logger.info(f"{self.namespace}嵌入库的idx2hash映射保存成功")
|
||||
|
||||
def load_from_file(self) -> None:
|
||||
"""从文件中加载"""
|
||||
if not os.path.exists(self.embedding_file_path):
|
||||
raise Exception(f"文件{self.embedding_file_path}不存在")
|
||||
logger.info("正在加载嵌入库...")
|
||||
logger.debug(f"正在从文件{self.embedding_file_path}中加载{self.namespace}嵌入库")
|
||||
data_frame = pd.read_parquet(self.embedding_file_path, engine="pyarrow")
|
||||
total = len(data_frame)
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TaskProgressColumn(),
|
||||
MofNCompleteColumn(),
|
||||
"•",
|
||||
TimeElapsedColumn(),
|
||||
"<",
|
||||
TimeRemainingColumn(),
|
||||
transient=False,
|
||||
) as progress:
|
||||
task = progress.add_task("加载嵌入库", total=total)
|
||||
for _, row in data_frame.iterrows():
|
||||
self.store[row["hash"]] = EmbeddingStoreItem(row["hash"], row["embedding"], row["str"])
|
||||
progress.update(task, advance=1)
|
||||
logger.info(f"{self.namespace}嵌入库加载成功")
|
||||
|
||||
try:
|
||||
if os.path.exists(self.index_file_path):
|
||||
logger.info(f"正在加载{self.namespace}嵌入库的FaissIndex...")
|
||||
logger.debug(f"正在从文件{self.index_file_path}中加载{self.namespace}嵌入库的FaissIndex")
|
||||
self.faiss_index = faiss.read_index(self.index_file_path)
|
||||
logger.info(f"{self.namespace}嵌入库的FaissIndex加载成功")
|
||||
else:
|
||||
raise Exception(f"文件{self.index_file_path}不存在")
|
||||
if os.path.exists(self.idx2hash_file_path):
|
||||
logger.info(f"正在加载{self.namespace}嵌入库的idx2hash映射...")
|
||||
logger.debug(f"正在从文件{self.idx2hash_file_path}中加载{self.namespace}嵌入库的idx2hash映射")
|
||||
with open(self.idx2hash_file_path, "r") as f:
|
||||
self.idx2hash = json.load(f)
|
||||
logger.info(f"{self.namespace}嵌入库的idx2hash映射加载成功")
|
||||
else:
|
||||
raise Exception(f"文件{self.idx2hash_file_path}不存在")
|
||||
except Exception as e:
|
||||
logger.error(f"加载{self.namespace}嵌入库的FaissIndex时发生错误：{e}")
|
||||
logger.warning("正在重建Faiss索引")
|
||||
self.build_faiss_index()
|
||||
logger.info(f"{self.namespace}嵌入库的FaissIndex重建成功")
|
||||
self.save_to_file()
|
||||
|
||||
def build_faiss_index(self) -> None:
|
||||
"""重新构建Faiss索引，以余弦相似度为度量"""
|
||||
# 获取所有的embedding
|
||||
array = []
|
||||
self.idx2hash = dict()
|
||||
for key in self.store:
|
||||
array.append(self.store[key].embedding)
|
||||
self.idx2hash[str(len(array) - 1)] = key
|
||||
embeddings = np.array(array, dtype=np.float32)
|
||||
# L2归一化
|
||||
faiss.normalize_L2(embeddings)
|
||||
# 构建索引
|
||||
self.faiss_index = faiss.IndexFlatIP(global_config.lpmm_knowledge.embedding_dimension)
|
||||
self.faiss_index.add(embeddings)
|
||||
|
||||
def search_top_k(self, query: List[float], k: int) -> List[Tuple[str, float]]:
|
||||
"""搜索最相似的k个项，以余弦相似度为度量
|
||||
Args:
|
||||
query: 查询的embedding
|
||||
k: 返回的最相似的k个项
|
||||
Returns:
|
||||
result: 最相似的k个项的(hash, 余弦相似度)列表
|
||||
"""
|
||||
if self.faiss_index is None:
|
||||
logger.debug("FaissIndex尚未构建,返回None")
|
||||
return []
|
||||
if self.idx2hash is None:
|
||||
logger.warning("idx2hash尚未构建,返回None")
|
||||
return []
|
||||
|
||||
# L2归一化
|
||||
faiss.normalize_L2(np.array([query], dtype=np.float32))
|
||||
# 搜索
|
||||
distances, indices = self.faiss_index.search(np.array([query]), k)
|
||||
# 整理结果
|
||||
indices = list(indices.flatten())
|
||||
distances = list(distances.flatten())
|
||||
result = [
|
||||
(self.idx2hash[str(int(idx))], float(sim))
|
||||
for (idx, sim) in zip(indices, distances, strict=False)
|
||||
if idx in range(len(self.idx2hash))
|
||||
]
|
||||
|
||||
return result
|
||||
|
||||
|
||||
class EmbeddingManager:
|
||||
def __init__(self, max_workers: int = DEFAULT_MAX_WORKERS, chunk_size: int = DEFAULT_CHUNK_SIZE):
|
||||
"""
|
||||
初始化EmbeddingManager
|
||||
|
||||
Args:
|
||||
max_workers: 最大线程数
|
||||
chunk_size: 每个线程处理的数据块大小
|
||||
"""
|
||||
self.paragraphs_embedding_store = EmbeddingStore(
|
||||
"paragraph", # type: ignore
|
||||
EMBEDDING_DATA_DIR_STR,
|
||||
max_workers=max_workers,
|
||||
chunk_size=chunk_size,
|
||||
)
|
||||
self.entities_embedding_store = EmbeddingStore(
|
||||
"entity", # type: ignore
|
||||
EMBEDDING_DATA_DIR_STR,
|
||||
max_workers=max_workers,
|
||||
chunk_size=chunk_size,
|
||||
)
|
||||
self.relation_embedding_store = EmbeddingStore(
|
||||
"relation", # type: ignore
|
||||
EMBEDDING_DATA_DIR_STR,
|
||||
max_workers=max_workers,
|
||||
chunk_size=chunk_size,
|
||||
)
|
||||
self.stored_pg_hashes = set()
|
||||
|
||||
def check_all_embedding_model_consistency(self):
|
||||
"""对所有嵌入库做模型一致性校验"""
|
||||
return self.paragraphs_embedding_store.check_embedding_model_consistency()
|
||||
|
||||
def _store_pg_into_embedding(self, raw_paragraphs: Dict[str, str]):
|
||||
"""将段落编码存入Embedding库"""
|
||||
self.paragraphs_embedding_store.batch_insert_strs(list(raw_paragraphs.values()), times=1)
|
||||
|
||||
def _store_ent_into_embedding(self, triple_list_data: Dict[str, List[List[str]]]):
|
||||
"""将实体编码存入Embedding库"""
|
||||
entities = set()
|
||||
for triple_list in triple_list_data.values():
|
||||
for triple in triple_list:
|
||||
entities.add(triple[0])
|
||||
entities.add(triple[2])
|
||||
self.entities_embedding_store.batch_insert_strs(list(entities), times=2)
|
||||
|
||||
def _store_rel_into_embedding(self, triple_list_data: Dict[str, List[List[str]]]):
|
||||
"""将关系编码存入Embedding库"""
|
||||
graph_triples = [] # a list of unique relation triple (in tuple) from all chunks
|
||||
for triples in triple_list_data.values():
|
||||
graph_triples.extend([tuple(t) for t in triples])
|
||||
graph_triples = list(set(graph_triples))
|
||||
self.relation_embedding_store.batch_insert_strs([str(triple) for triple in graph_triples], times=3)
|
||||
|
||||
def load_from_file(self):
|
||||
"""从文件加载"""
|
||||
self.paragraphs_embedding_store.load_from_file()
|
||||
self.entities_embedding_store.load_from_file()
|
||||
self.relation_embedding_store.load_from_file()
|
||||
# 从段落库中获取已存储的hash
|
||||
self.stored_pg_hashes = set(self.paragraphs_embedding_store.store.keys())
|
||||
|
||||
def store_new_data_set(
|
||||
self,
|
||||
raw_paragraphs: Dict[str, str],
|
||||
triple_list_data: Dict[str, List[List[str]]],
|
||||
):
|
||||
if not self.check_all_embedding_model_consistency():
|
||||
raise Exception("嵌入模型与本地存储不一致，请检查模型设置或清空嵌入库后重试。")
|
||||
"""存储新的数据集"""
|
||||
self._store_pg_into_embedding(raw_paragraphs)
|
||||
self._store_ent_into_embedding(triple_list_data)
|
||||
self._store_rel_into_embedding(triple_list_data)
|
||||
self.stored_pg_hashes.update(raw_paragraphs.keys())
|
||||
|
||||
def save_to_file(self):
|
||||
"""保存到文件"""
|
||||
self.paragraphs_embedding_store.save_to_file()
|
||||
self.entities_embedding_store.save_to_file()
|
||||
self.relation_embedding_store.save_to_file()
|
||||
|
||||
def rebuild_faiss_index(self):
|
||||
"""重建Faiss索引（请在添加新数据后调用）"""
|
||||
self.paragraphs_embedding_store.build_faiss_index()
|
||||
self.entities_embedding_store.build_faiss_index()
|
||||
self.relation_embedding_store.build_faiss_index()
|
||||
5
src/chat/knowledge/global_logger.py
Normal file
5
src/chat/knowledge/global_logger.py
Normal file
@@ -0,0 +1,5 @@
|
||||
# Configure logger
|
||||
|
||||
from src.common.logger import get_logger
|
||||
|
||||
logger = get_logger("lpmm")
|
||||
175
src/chat/knowledge/ie_process.py
Normal file
175
src/chat/knowledge/ie_process.py
Normal file
@@ -0,0 +1,175 @@
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
from typing import List, Union
|
||||
|
||||
from .global_logger import logger
|
||||
from . import prompt_template
|
||||
from .knowledge_lib import INVALID_ENTITY
|
||||
from src.llm_models.utils_model import LLMRequest
|
||||
from json_repair import repair_json
|
||||
|
||||
|
||||
def _extract_json_from_text(text: str):
|
||||
# sourcery skip: assign-if-exp, extract-method
|
||||
"""从文本中提取JSON数据的高容错方法"""
|
||||
if text is None:
|
||||
logger.error("输入文本为None")
|
||||
return []
|
||||
|
||||
try:
|
||||
fixed_json = repair_json(text)
|
||||
if isinstance(fixed_json, str):
|
||||
parsed_json = json.loads(fixed_json)
|
||||
else:
|
||||
parsed_json = fixed_json
|
||||
|
||||
# 如果是列表，直接返回
|
||||
if isinstance(parsed_json, list):
|
||||
return parsed_json
|
||||
|
||||
# 如果是字典且只有一个项目，可能包装了列表
|
||||
if isinstance(parsed_json, dict):
|
||||
# 如果字典只有一个键，并且值是列表，返回那个列表
|
||||
if len(parsed_json) == 1:
|
||||
value = list(parsed_json.values())[0]
|
||||
if isinstance(value, list):
|
||||
return value
|
||||
return parsed_json
|
||||
|
||||
# 其他情况，尝试转换为列表
|
||||
logger.warning(f"解析的JSON不是预期格式: {type(parsed_json)}, 内容: {parsed_json}")
|
||||
return []
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"JSON提取失败: {e}, 原始文本: {text[:100] if text else 'None'}...")
|
||||
return []
|
||||
|
||||
|
||||
def _entity_extract(llm_req: LLMRequest, paragraph: str) -> List[str]:
|
||||
# sourcery skip: reintroduce-else, swap-if-else-branches, use-named-expression
|
||||
"""对段落进行实体提取，返回提取出的实体列表（JSON格式）"""
|
||||
entity_extract_context = prompt_template.build_entity_extract_context(paragraph)
|
||||
|
||||
# 使用 asyncio.run 来运行异步方法
|
||||
try:
|
||||
# 如果当前已有事件循环在运行，使用它
|
||||
loop = asyncio.get_running_loop()
|
||||
future = asyncio.run_coroutine_threadsafe(llm_req.generate_response_async(entity_extract_context), loop)
|
||||
response, _ = future.result()
|
||||
except RuntimeError:
|
||||
# 如果没有运行中的事件循环，直接使用 asyncio.run
|
||||
response, _ = asyncio.run(llm_req.generate_response_async(entity_extract_context))
|
||||
|
||||
# 添加调试日志
|
||||
logger.debug(f"LLM返回的原始响应: {response}")
|
||||
|
||||
entity_extract_result = _extract_json_from_text(response)
|
||||
|
||||
# 检查返回的是否为有效的实体列表
|
||||
if not isinstance(entity_extract_result, list):
|
||||
if not isinstance(entity_extract_result, dict):
|
||||
raise ValueError(f"实体提取结果格式错误，期望列表但得到: {type(entity_extract_result)}")
|
||||
|
||||
# 尝试常见的键名
|
||||
for key in ["entities", "result", "data", "items"]:
|
||||
if key in entity_extract_result and isinstance(entity_extract_result[key], list):
|
||||
entity_extract_result = entity_extract_result[key]
|
||||
break
|
||||
else:
|
||||
# 如果找不到合适的列表，抛出异常
|
||||
raise ValueError(f"实体提取结果格式错误，期望列表但得到: {type(entity_extract_result)}")
|
||||
# 过滤无效实体
|
||||
entity_extract_result = [
|
||||
entity
|
||||
for entity in entity_extract_result
|
||||
if (entity is not None) and (entity != "") and (entity not in INVALID_ENTITY)
|
||||
]
|
||||
|
||||
if not entity_extract_result:
|
||||
raise ValueError("实体提取结果为空")
|
||||
|
||||
return entity_extract_result
|
||||
|
||||
|
||||
def _rdf_triple_extract(llm_req: LLMRequest, paragraph: str, entities: list) -> List[List[str]]:
|
||||
"""对段落进行实体提取，返回提取出的实体列表（JSON格式）"""
|
||||
rdf_extract_context = prompt_template.build_rdf_triple_extract_context(
|
||||
paragraph, entities=json.dumps(entities, ensure_ascii=False)
|
||||
)
|
||||
|
||||
# 使用 asyncio.run 来运行异步方法
|
||||
try:
|
||||
# 如果当前已有事件循环在运行，使用它
|
||||
loop = asyncio.get_running_loop()
|
||||
future = asyncio.run_coroutine_threadsafe(llm_req.generate_response_async(rdf_extract_context), loop)
|
||||
response, _ = future.result()
|
||||
except RuntimeError:
|
||||
# 如果没有运行中的事件循环，直接使用 asyncio.run
|
||||
response, _ = asyncio.run(llm_req.generate_response_async(rdf_extract_context))
|
||||
|
||||
# 添加调试日志
|
||||
logger.debug(f"RDF LLM返回的原始响应: {response}")
|
||||
|
||||
rdf_triple_result = _extract_json_from_text(response)
|
||||
|
||||
# 检查返回的是否为有效的三元组列表
|
||||
if not isinstance(rdf_triple_result, list):
|
||||
if not isinstance(rdf_triple_result, dict):
|
||||
raise ValueError(f"RDF三元组提取结果格式错误，期望列表但得到: {type(rdf_triple_result)}")
|
||||
|
||||
# 尝试常见的键名
|
||||
for key in ["triples", "result", "data", "items"]:
|
||||
if key in rdf_triple_result and isinstance(rdf_triple_result[key], list):
|
||||
rdf_triple_result = rdf_triple_result[key]
|
||||
break
|
||||
else:
|
||||
# 如果找不到合适的列表，抛出异常
|
||||
raise ValueError(f"RDF三元组提取结果格式错误，期望列表但得到: {type(rdf_triple_result)}")
|
||||
# 验证三元组格式
|
||||
for triple in rdf_triple_result:
|
||||
if (
|
||||
not isinstance(triple, list)
|
||||
or len(triple) != 3
|
||||
or (triple[0] is None or triple[1] is None or triple[2] is None)
|
||||
or "" in triple
|
||||
):
|
||||
raise ValueError("RDF提取结果格式错误")
|
||||
|
||||
return rdf_triple_result
|
||||
|
||||
|
||||
def info_extract_from_str(
|
||||
llm_client_for_ner: LLMRequest, llm_client_for_rdf: LLMRequest, paragraph: str
|
||||
) -> Union[tuple[None, None], tuple[list[str], list[list[str]]]]:
|
||||
try_count = 0
|
||||
while True:
|
||||
try:
|
||||
entity_extract_result = _entity_extract(llm_client_for_ner, paragraph)
|
||||
break
|
||||
except Exception as e:
|
||||
logger.warning(f"实体提取失败，错误信息：{e}")
|
||||
try_count += 1
|
||||
if try_count < 3:
|
||||
logger.warning("将于5秒后重试")
|
||||
time.sleep(5)
|
||||
else:
|
||||
logger.error("实体提取失败，已达最大重试次数")
|
||||
return None, None
|
||||
|
||||
try_count = 0
|
||||
while True:
|
||||
try:
|
||||
rdf_triple_extract_result = _rdf_triple_extract(llm_client_for_rdf, paragraph, entity_extract_result)
|
||||
break
|
||||
except Exception as e:
|
||||
logger.warning(f"实体提取失败，错误信息：{e}")
|
||||
try_count += 1
|
||||
if try_count < 3:
|
||||
logger.warning("将于5秒后重试")
|
||||
time.sleep(5)
|
||||
else:
|
||||
logger.error("实体提取失败，已达最大重试次数")
|
||||
return None, None
|
||||
|
||||
return entity_extract_result, rdf_triple_extract_result
|
||||
438
src/chat/knowledge/kg_manager.py
Normal file
438
src/chat/knowledge/kg_manager.py
Normal file
@@ -0,0 +1,438 @@
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from typing import Dict, List, Tuple
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from rich.progress import (
|
||||
Progress,
|
||||
BarColumn,
|
||||
TimeElapsedColumn,
|
||||
TimeRemainingColumn,
|
||||
TaskProgressColumn,
|
||||
MofNCompleteColumn,
|
||||
SpinnerColumn,
|
||||
TextColumn,
|
||||
)
|
||||
from quick_algo import di_graph, pagerank
|
||||
|
||||
|
||||
from .utils.hash import get_sha256
|
||||
from .embedding_store import EmbeddingManager, EmbeddingStoreItem
|
||||
from src.config.config import global_config
|
||||
|
||||
from .global_logger import logger
|
||||
|
||||
|
||||
def _get_kg_dir():
|
||||
"""
|
||||
安全地获取KG数据目录路径
|
||||
"""
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
root_path: str = os.path.abspath(os.path.join(current_dir, "..", "..", ".."))
|
||||
kg_dir = os.path.join(root_path, "data/rag")
|
||||
|
||||
return str(kg_dir).replace("\\", "/")
|
||||
|
||||
|
||||
# 延迟初始化，避免在模块加载时就访问可能未初始化的 local_storage
|
||||
def get_kg_dir_str():
|
||||
"""获取KG目录字符串"""
|
||||
return _get_kg_dir()
|
||||
|
||||
|
||||
class KGManager:
|
||||
def __init__(self):
|
||||
# 会被保存的字段
|
||||
# 存储段落的hash值，用于去重
|
||||
self.stored_paragraph_hashes = set()
|
||||
# 实体出现次数
|
||||
self.ent_appear_cnt = {}
|
||||
# KG
|
||||
self.graph = di_graph.DiGraph()
|
||||
|
||||
# 持久化相关 - 使用延迟初始化的路径
|
||||
self.dir_path = get_kg_dir_str()
|
||||
self.graph_data_path = self.dir_path + "/" + "rag-graph" + ".graphml"
|
||||
self.ent_cnt_data_path = self.dir_path + "/" + "rag-ent-cnt" + ".parquet"
|
||||
self.pg_hash_file_path = self.dir_path + "/" + "rag-pg-hash" + ".json"
|
||||
|
||||
def save_to_file(self):
|
||||
"""将KG数据保存到文件"""
|
||||
# 确保目录存在
|
||||
if not os.path.exists(self.dir_path):
|
||||
os.makedirs(self.dir_path, exist_ok=True)
|
||||
|
||||
# 保存KG
|
||||
di_graph.save_to_file(self.graph, self.graph_data_path)
|
||||
|
||||
# 保存实体计数到文件
|
||||
ent_cnt_df = pd.DataFrame([{"hash_key": k, "appear_cnt": v} for k, v in self.ent_appear_cnt.items()])
|
||||
ent_cnt_df.to_parquet(self.ent_cnt_data_path, engine="pyarrow", index=False)
|
||||
|
||||
# 保存段落hash到文件
|
||||
with open(self.pg_hash_file_path, "w", encoding="utf-8") as f:
|
||||
data = {"stored_paragraph_hashes": list(self.stored_paragraph_hashes)}
|
||||
f.write(json.dumps(data, ensure_ascii=False, indent=4))
|
||||
|
||||
def load_from_file(self):
|
||||
"""从文件加载KG数据"""
|
||||
# 确保文件存在
|
||||
if not os.path.exists(self.pg_hash_file_path):
|
||||
raise FileNotFoundError(f"KG段落hash文件{self.pg_hash_file_path}不存在")
|
||||
if not os.path.exists(self.ent_cnt_data_path):
|
||||
raise FileNotFoundError(f"KG实体计数文件{self.ent_cnt_data_path}不存在")
|
||||
if not os.path.exists(self.graph_data_path):
|
||||
raise FileNotFoundError(f"KG图文件{self.graph_data_path}不存在")
|
||||
|
||||
# 加载段落hash
|
||||
with open(self.pg_hash_file_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
self.stored_paragraph_hashes = set(data["stored_paragraph_hashes"])
|
||||
|
||||
# 加载实体计数
|
||||
ent_cnt_df = pd.read_parquet(self.ent_cnt_data_path, engine="pyarrow")
|
||||
self.ent_appear_cnt = dict({row["hash_key"]: row["appear_cnt"] for _, row in ent_cnt_df.iterrows()})
|
||||
|
||||
# 加载KG
|
||||
self.graph = di_graph.load_from_file(self.graph_data_path)
|
||||
|
||||
def _build_edges_between_ent(
|
||||
self,
|
||||
node_to_node: Dict[Tuple[str, str], float],
|
||||
triple_list_data: Dict[str, List[List[str]]],
|
||||
):
|
||||
"""构建实体节点之间的关系，同时统计实体出现次数"""
|
||||
for triple_list in triple_list_data.values():
|
||||
entity_set = set()
|
||||
for triple in triple_list:
|
||||
if triple[0] == triple[2]:
|
||||
# 避免自连接
|
||||
continue
|
||||
# 一个triple就是一条边（同时构建双向联系）
|
||||
hash_key1 = "entity" + "-" + get_sha256(triple[0])
|
||||
hash_key2 = "entity" + "-" + get_sha256(triple[2])
|
||||
node_to_node[(hash_key1, hash_key2)] = node_to_node.get((hash_key1, hash_key2), 0) + 1.0
|
||||
node_to_node[(hash_key2, hash_key1)] = node_to_node.get((hash_key2, hash_key1), 0) + 1.0
|
||||
entity_set.add(hash_key1)
|
||||
entity_set.add(hash_key2)
|
||||
|
||||
# 实体出现次数统计
|
||||
for hash_key in entity_set:
|
||||
self.ent_appear_cnt[hash_key] = self.ent_appear_cnt.get(hash_key, 0) + 1.0
|
||||
|
||||
@staticmethod
|
||||
def _build_edges_between_ent_pg(
|
||||
node_to_node: Dict[Tuple[str, str], float],
|
||||
triple_list_data: Dict[str, List[List[str]]],
|
||||
):
|
||||
"""构建实体节点与文段节点之间的关系"""
|
||||
for idx in triple_list_data:
|
||||
for triple in triple_list_data[idx]:
|
||||
ent_hash_key = "entity" + "-" + get_sha256(triple[0])
|
||||
pg_hash_key = "paragraph" + "-" + str(idx)
|
||||
node_to_node[(ent_hash_key, pg_hash_key)] = node_to_node.get((ent_hash_key, pg_hash_key), 0) + 1.0
|
||||
|
||||
@staticmethod
|
||||
def _synonym_connect(
|
||||
node_to_node: Dict[Tuple[str, str], float],
|
||||
triple_list_data: Dict[str, List[List[str]]],
|
||||
embedding_manager: EmbeddingManager,
|
||||
) -> int:
|
||||
"""同义词连接"""
|
||||
new_edge_cnt = 0
|
||||
# 获取所有实体节点的hash值
|
||||
ent_hash_list = set()
|
||||
for triple_list in triple_list_data.values():
|
||||
for triple in triple_list:
|
||||
ent_hash_list.add("entity" + "-" + get_sha256(triple[0]))
|
||||
ent_hash_list.add("entity" + "-" + get_sha256(triple[2]))
|
||||
ent_hash_list = list(ent_hash_list)
|
||||
|
||||
synonym_hash_set = set()
|
||||
synonym_result = {}
|
||||
|
||||
# rich 进度条
|
||||
total = len(ent_hash_list)
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TaskProgressColumn(),
|
||||
MofNCompleteColumn(),
|
||||
"•",
|
||||
TimeElapsedColumn(),
|
||||
"<",
|
||||
TimeRemainingColumn(),
|
||||
transient=False,
|
||||
) as progress:
|
||||
task = progress.add_task("同义词连接", total=total)
|
||||
for ent_hash in ent_hash_list:
|
||||
if ent_hash in synonym_hash_set:
|
||||
progress.update(task, advance=1)
|
||||
continue
|
||||
ent = embedding_manager.entities_embedding_store.store.get(ent_hash)
|
||||
if ent is None:
|
||||
progress.update(task, advance=1)
|
||||
continue
|
||||
assert isinstance(ent, EmbeddingStoreItem)
|
||||
# 查询相似实体
|
||||
similar_ents = embedding_manager.entities_embedding_store.search_top_k(
|
||||
ent.embedding, global_config.lpmm_knowledge.rag_synonym_search_top_k
|
||||
)
|
||||
res_ent = [] # Debug
|
||||
for res_ent_hash, similarity in similar_ents:
|
||||
if res_ent_hash == ent_hash:
|
||||
# 避免自连接
|
||||
continue
|
||||
if similarity < global_config.lpmm_knowledge.rag_synonym_threshold:
|
||||
# 相似度阈值
|
||||
continue
|
||||
node_to_node[(res_ent_hash, ent_hash)] = similarity
|
||||
node_to_node[(ent_hash, res_ent_hash)] = similarity
|
||||
synonym_hash_set.add(res_ent_hash)
|
||||
new_edge_cnt += 1
|
||||
res_ent.append(
|
||||
(
|
||||
embedding_manager.entities_embedding_store.store[res_ent_hash].str,
|
||||
similarity,
|
||||
)
|
||||
) # Debug
|
||||
synonym_result[ent.str] = res_ent
|
||||
progress.update(task, advance=1)
|
||||
|
||||
for k, v in synonym_result.items():
|
||||
print(f'"{k}"的相似实体为：{v}')
|
||||
return new_edge_cnt
|
||||
|
||||
def _update_graph(
|
||||
self,
|
||||
node_to_node: Dict[Tuple[str, str], float],
|
||||
embedding_manager: EmbeddingManager,
|
||||
):
|
||||
"""更新KG图结构
|
||||
|
||||
流程：
|
||||
1. 更新图结构：遍历所有待添加的新边
|
||||
- 若是新边，则添加到图中
|
||||
- 若是已存在的边，则更新边的权重
|
||||
2. 更新新节点的属性
|
||||
"""
|
||||
existed_nodes = self.graph.get_node_list()
|
||||
existed_edges = [str((edge[0], edge[1])) for edge in self.graph.get_edge_list()]
|
||||
|
||||
now_time = time.time()
|
||||
|
||||
# 更新图结构
|
||||
for src_tgt, weight in node_to_node.items():
|
||||
key = str(src_tgt)
|
||||
# 检查边是否已存在
|
||||
if key not in existed_edges:
|
||||
# 新边
|
||||
self.graph.add_edge(
|
||||
di_graph.DiEdge(
|
||||
src_tgt[0],
|
||||
src_tgt[1],
|
||||
{
|
||||
"weight": weight,
|
||||
"create_time": now_time,
|
||||
"update_time": now_time,
|
||||
},
|
||||
)
|
||||
)
|
||||
else:
|
||||
# 已存在的边
|
||||
edge_item = self.graph[src_tgt[0], src_tgt[1]]
|
||||
edge_item["weight"] += weight
|
||||
edge_item["update_time"] = now_time
|
||||
self.graph.update_edge(edge_item)
|
||||
|
||||
# 更新新节点属性
|
||||
for src_tgt in node_to_node.keys():
|
||||
for node_hash in src_tgt:
|
||||
if node_hash not in existed_nodes:
|
||||
if node_hash.startswith("entity"):
|
||||
# 新增实体节点
|
||||
node = embedding_manager.entities_embedding_store.store.get(node_hash)
|
||||
if node is None:
|
||||
logger.warning(f"实体节点 {node_hash} 在嵌入库中不存在，跳过")
|
||||
continue
|
||||
assert isinstance(node, EmbeddingStoreItem)
|
||||
node_item = self.graph[node_hash]
|
||||
node_item["content"] = node.str
|
||||
node_item["type"] = "ent"
|
||||
node_item["create_time"] = now_time
|
||||
self.graph.update_node(node_item)
|
||||
elif node_hash.startswith("paragraph"):
|
||||
# 新增文段节点
|
||||
node = embedding_manager.paragraphs_embedding_store.store.get(node_hash)
|
||||
if node is None:
|
||||
logger.warning(f"段落节点 {node_hash} 在嵌入库中不存在，跳过")
|
||||
continue
|
||||
assert isinstance(node, EmbeddingStoreItem)
|
||||
content = node.str.replace("\n", " ")
|
||||
node_item = self.graph[node_hash]
|
||||
node_item["content"] = content if len(content) < 8 else content[:8] + "..."
|
||||
node_item["type"] = "pg"
|
||||
node_item["create_time"] = now_time
|
||||
self.graph.update_node(node_item)
|
||||
|
||||
def build_kg(
|
||||
self,
|
||||
triple_list_data: Dict[str, List[List[str]]],
|
||||
embedding_manager: EmbeddingManager,
|
||||
):
|
||||
"""增量式构建KG
|
||||
|
||||
注意：应当在调用该方法后保存KG
|
||||
|
||||
Args:
|
||||
triple_list_data: 三元组数据
|
||||
embedding_manager: EmbeddingManager对象
|
||||
"""
|
||||
# 实体之间的联系
|
||||
node_to_node = dict()
|
||||
|
||||
# 构建实体节点之间的关系，同时统计实体出现次数
|
||||
logger.info("正在构建KG实体节点之间的关系，同时统计实体出现次数")
|
||||
# 从三元组提取实体对
|
||||
self._build_edges_between_ent(node_to_node, triple_list_data)
|
||||
|
||||
# 构建实体节点与文段节点之间的关系
|
||||
logger.info("正在构建KG实体节点与文段节点之间的关系")
|
||||
self._build_edges_between_ent_pg(node_to_node, triple_list_data)
|
||||
|
||||
# 近义词扩展链接
|
||||
# 对每个实体节点，找到最相似的实体节点，建立扩展连接
|
||||
logger.info("正在进行近义词扩展链接")
|
||||
self._synonym_connect(node_to_node, triple_list_data, embedding_manager)
|
||||
|
||||
# 构建图
|
||||
self._update_graph(node_to_node, embedding_manager)
|
||||
|
||||
# 记录已处理（存储）的段落hash
|
||||
for idx in triple_list_data:
|
||||
self.stored_paragraph_hashes.add(str(idx))
|
||||
|
||||
def kg_search(
|
||||
self,
|
||||
relation_search_result: List[Tuple[Tuple[str, str, str], float]],
|
||||
paragraph_search_result: List[Tuple[str, float]],
|
||||
embed_manager: EmbeddingManager,
|
||||
):
|
||||
"""RAG搜索与PageRank
|
||||
|
||||
Args:
|
||||
relation_search_result: RelationEmbedding的搜索结果（relation_tripple, similarity）
|
||||
paragraph_search_result: ParagraphEmbedding的搜索结果（paragraph_hash, similarity）
|
||||
embed_manager: EmbeddingManager对象
|
||||
"""
|
||||
# 图中存在的节点总集
|
||||
existed_nodes = self.graph.get_node_list()
|
||||
|
||||
# 准备PPR使用的数据
|
||||
# 节点权重：实体
|
||||
ent_weights = {}
|
||||
# 节点权重：文段
|
||||
pg_weights = {}
|
||||
|
||||
# 以下部分处理实体权重ent_weights
|
||||
|
||||
# 针对每个关系，提取出其中的主宾短语作为两个实体，并记录对应的三元组的相似度作为权重依据
|
||||
ent_sim_scores = {}
|
||||
for relation_hash, similarity, _ in relation_search_result:
|
||||
# 提取主宾短语
|
||||
relation = embed_manager.relation_embedding_store.store.get(relation_hash).str
|
||||
assert relation is not None # 断言：relation不为空
|
||||
# 关系三元组
|
||||
triple = relation[2:-2].split("', '")
|
||||
for ent in [(triple[0]), (triple[2])]:
|
||||
ent_hash = "entity" + "-" + get_sha256(ent)
|
||||
if ent_hash in existed_nodes: # 该实体需在KG中存在
|
||||
if ent_hash not in ent_sim_scores: # 尚未记录的实体
|
||||
ent_sim_scores[ent_hash] = []
|
||||
ent_sim_scores[ent_hash].append(similarity)
|
||||
|
||||
ent_mean_scores = {} # 记录实体的平均相似度
|
||||
for ent_hash, scores in ent_sim_scores.items():
|
||||
# 先对相似度进行累加，然后与实体计数相除获取最终权重
|
||||
ent_weights[ent_hash] = float(np.sum(scores)) / self.ent_appear_cnt[ent_hash]
|
||||
# 记录实体的平均相似度，用于后续的top_k筛选
|
||||
ent_mean_scores[ent_hash] = float(np.mean(scores))
|
||||
del ent_sim_scores
|
||||
|
||||
ent_weights_max = max(ent_weights.values())
|
||||
ent_weights_min = min(ent_weights.values())
|
||||
if ent_weights_max == ent_weights_min:
|
||||
# 只有一个相似度，则全赋值为1
|
||||
for ent_hash in ent_weights.keys():
|
||||
ent_weights[ent_hash] = 1.0
|
||||
else:
|
||||
down_edge = global_config.lpmm_knowledge.qa_paragraph_node_weight
|
||||
# 缩放取值区间至[down_edge, 1]
|
||||
for ent_hash, score in ent_weights.items():
|
||||
# 缩放相似度
|
||||
ent_weights[ent_hash] = (
|
||||
(score - ent_weights_min) * (1 - down_edge) / (ent_weights_max - ent_weights_min)
|
||||
) + down_edge
|
||||
|
||||
# 取平均相似度的top_k实体
|
||||
top_k = global_config.lpmm_knowledge.qa_ent_filter_top_k
|
||||
if len(ent_mean_scores) > top_k:
|
||||
# 从大到小排序，取后len - k个
|
||||
ent_mean_scores = {k: v for k, v in sorted(ent_mean_scores.items(), key=lambda item: item[1], reverse=True)}
|
||||
for ent_hash, _ in ent_mean_scores.items():
|
||||
# 删除被淘汰的实体节点权重设置
|
||||
del ent_weights[ent_hash]
|
||||
del top_k, ent_mean_scores
|
||||
|
||||
# 以下部分处理文段权重pg_weights
|
||||
|
||||
# 将搜索结果中文段的相似度归一化作为权重
|
||||
pg_sim_scores = {}
|
||||
pg_sim_score_max = 0.0
|
||||
pg_sim_score_min = 1.0
|
||||
for pg_hash, similarity in paragraph_search_result:
|
||||
# 查找最大和最小值
|
||||
pg_sim_score_max = max(pg_sim_score_max, similarity)
|
||||
pg_sim_score_min = min(pg_sim_score_min, similarity)
|
||||
pg_sim_scores[pg_hash] = similarity
|
||||
|
||||
# 归一化
|
||||
for pg_hash, similarity in pg_sim_scores.items():
|
||||
# 归一化相似度
|
||||
pg_sim_scores[pg_hash] = (similarity - pg_sim_score_min) / (pg_sim_score_max - pg_sim_score_min)
|
||||
del pg_sim_score_max, pg_sim_score_min
|
||||
|
||||
for pg_hash, score in pg_sim_scores.items():
|
||||
pg_weights[pg_hash] = (
|
||||
score * global_config.lpmm_knowledge.qa_paragraph_node_weight
|
||||
) # 文段权重 = 归一化相似度 * 文段节点权重参数
|
||||
del pg_sim_scores
|
||||
|
||||
# 最终权重数据 = 实体权重 + 文段权重
|
||||
ppr_node_weights = {k: v for d in [ent_weights, pg_weights] for k, v in d.items()}
|
||||
del ent_weights, pg_weights
|
||||
|
||||
# PersonalizedPageRank
|
||||
ppr_res = pagerank.run_pagerank(
|
||||
self.graph,
|
||||
personalization=ppr_node_weights,
|
||||
max_iter=100,
|
||||
alpha=global_config.lpmm_knowledge.qa_ppr_damping,
|
||||
)
|
||||
|
||||
# 获取最终结果
|
||||
# 从搜索结果中提取文段节点的结果
|
||||
passage_node_res = [
|
||||
(node_key, score)
|
||||
for node_key, score in ppr_res.items()
|
||||
if node_key.startswith("paragraph")
|
||||
]
|
||||
del ppr_res
|
||||
|
||||
# 排序：按照分数从大到小
|
||||
passage_node_res = sorted(passage_node_res, key=lambda item: item[1], reverse=True)
|
||||
|
||||
return passage_node_res, ppr_node_weights
|
||||
79
src/chat/knowledge/knowledge_lib.py
Normal file
79
src/chat/knowledge/knowledge_lib.py
Normal file
@@ -0,0 +1,79 @@
|
||||
from src.chat.knowledge.embedding_store import EmbeddingManager
|
||||
from src.chat.knowledge.qa_manager import QAManager
|
||||
from src.chat.knowledge.kg_manager import KGManager
|
||||
from src.chat.knowledge.global_logger import logger
|
||||
from src.config.config import global_config
|
||||
import os
|
||||
|
||||
INVALID_ENTITY = [
|
||||
"",
|
||||
"你",
|
||||
"他",
|
||||
"她",
|
||||
"它",
|
||||
"我们",
|
||||
"你们",
|
||||
"他们",
|
||||
"她们",
|
||||
"它们",
|
||||
]
|
||||
|
||||
RAG_GRAPH_NAMESPACE = "rag-graph"
|
||||
RAG_ENT_CNT_NAMESPACE = "rag-ent-cnt"
|
||||
RAG_PG_HASH_NAMESPACE = "rag-pg-hash"
|
||||
|
||||
|
||||
ROOT_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", ".."))
|
||||
DATA_PATH = os.path.join(ROOT_PATH, "data")
|
||||
|
||||
|
||||
qa_manager = None
|
||||
inspire_manager = None
|
||||
|
||||
# 检查LPMM知识库是否启用
|
||||
if global_config.lpmm_knowledge.enable:
|
||||
logger.info("正在初始化Mai-LPMM")
|
||||
logger.info("创建LLM客户端")
|
||||
|
||||
# 初始化Embedding库
|
||||
embed_manager = EmbeddingManager()
|
||||
logger.info("正在从文件加载Embedding库")
|
||||
try:
|
||||
embed_manager.load_from_file()
|
||||
except Exception as e:
|
||||
logger.warning(f"此消息不会影响正常使用：从文件加载Embedding库时，{e}")
|
||||
# logger.warning("如果你是第一次导入知识，或者还未导入知识，请忽略此错误")
|
||||
logger.info("Embedding库加载完成")
|
||||
# 初始化KG
|
||||
kg_manager = KGManager()
|
||||
logger.info("正在从文件加载KG")
|
||||
try:
|
||||
kg_manager.load_from_file()
|
||||
except Exception as e:
|
||||
logger.warning(f"此消息不会影响正常使用：从文件加载KG时，{e}")
|
||||
# logger.warning("如果你是第一次导入知识，或者还未导入知识，请忽略此错误")
|
||||
logger.info("KG加载完成")
|
||||
|
||||
logger.info(f"KG节点数量：{len(kg_manager.graph.get_node_list())}")
|
||||
logger.info(f"KG边数量：{len(kg_manager.graph.get_edge_list())}")
|
||||
|
||||
# 数据比对：Embedding库与KG的段落hash集合
|
||||
for pg_hash in kg_manager.stored_paragraph_hashes:
|
||||
key = f"paragraph-{pg_hash}"
|
||||
if key not in embed_manager.stored_pg_hashes:
|
||||
logger.warning(f"KG中存在Embedding库中不存在的段落：{key}")
|
||||
|
||||
# 问答系统（用于知识库）
|
||||
qa_manager = QAManager(
|
||||
embed_manager,
|
||||
kg_manager,
|
||||
)
|
||||
|
||||
# # 记忆激活（用于记忆库）
|
||||
# inspire_manager = MemoryActiveManager(
|
||||
# embed_manager,
|
||||
# llm_client_list[global_config["embedding"]["provider"]],
|
||||
# )
|
||||
else:
|
||||
logger.info("LPMM知识库已禁用，跳过初始化")
|
||||
# 创建空的占位符对象，避免导入错误
|
||||
154
src/chat/knowledge/open_ie.py
Normal file
154
src/chat/knowledge/open_ie.py
Normal file
@@ -0,0 +1,154 @@
|
||||
import json
|
||||
import os
|
||||
import glob
|
||||
from typing import Any, Dict, List
|
||||
|
||||
|
||||
from .knowledge_lib import INVALID_ENTITY, ROOT_PATH, DATA_PATH
|
||||
# from src.manager.local_store_manager import local_storage
|
||||
|
||||
|
||||
def _filter_invalid_entities(entities: List[str]) -> List[str]:
|
||||
"""过滤无效的实体"""
|
||||
valid_entities = set()
|
||||
for entity in entities:
|
||||
if not isinstance(entity, str) or entity.strip() == "" or entity in INVALID_ENTITY or entity in valid_entities:
|
||||
# 非字符串/空字符串/在无效实体列表中/重复
|
||||
continue
|
||||
valid_entities.add(entity)
|
||||
|
||||
return list(valid_entities)
|
||||
|
||||
|
||||
def _filter_invalid_triples(triples: List[List[str]]) -> List[List[str]]:
|
||||
"""过滤无效的三元组"""
|
||||
unique_triples = set()
|
||||
valid_triples = []
|
||||
|
||||
for triple in triples:
|
||||
if len(triple) != 3 or (
|
||||
(not isinstance(triple[0], str) or triple[0].strip() == "")
|
||||
or (not isinstance(triple[1], str) or triple[1].strip() == "")
|
||||
or (not isinstance(triple[2], str) or triple[2].strip() == "")
|
||||
):
|
||||
# 三元组长度不为3，或其中存在空值
|
||||
continue
|
||||
|
||||
valid_triple = [str(item) for item in triple]
|
||||
if tuple(valid_triple) not in unique_triples:
|
||||
unique_triples.add(tuple(valid_triple))
|
||||
valid_triples.append(valid_triple)
|
||||
|
||||
return valid_triples
|
||||
|
||||
|
||||
class OpenIE:
|
||||
"""
|
||||
OpenIE规约的数据格式为如下
|
||||
{
|
||||
"docs": [
|
||||
{
|
||||
"idx": "文档的唯一标识符（通常是文本的SHA256哈希值）",
|
||||
"passage": "文档的原始文本",
|
||||
"extracted_entities": ["实体1", "实体2", ...],
|
||||
"extracted_triples": [["主语", "谓语", "宾语"], ...]
|
||||
},
|
||||
...
|
||||
],
|
||||
"avg_ent_chars": "实体平均字符数",
|
||||
"avg_ent_words": "实体平均词数"
|
||||
}
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
docs: List[Dict[str, Any]],
|
||||
avg_ent_chars,
|
||||
avg_ent_words,
|
||||
):
|
||||
self.docs = docs
|
||||
self.avg_ent_chars = avg_ent_chars
|
||||
self.avg_ent_words = avg_ent_words
|
||||
|
||||
for doc in self.docs:
|
||||
# 过滤实体列表
|
||||
doc["extracted_entities"] = _filter_invalid_entities(doc["extracted_entities"])
|
||||
# 过滤无效的三元组
|
||||
doc["extracted_triples"] = _filter_invalid_triples(doc["extracted_triples"])
|
||||
|
||||
@staticmethod
|
||||
def _from_dict(data_list):
|
||||
"""从多个字典合并OpenIE对象"""
|
||||
# data_list: List[dict]
|
||||
all_docs = []
|
||||
for data in data_list:
|
||||
all_docs.extend(data.get("docs", []))
|
||||
# 重新计算统计
|
||||
sum_phrase_chars = sum([len(e) for chunk in all_docs for e in chunk["extracted_entities"]])
|
||||
sum_phrase_words = sum([len(e.split()) for chunk in all_docs for e in chunk["extracted_entities"]])
|
||||
num_phrases = sum([len(chunk["extracted_entities"]) for chunk in all_docs])
|
||||
avg_ent_chars = round(sum_phrase_chars / num_phrases, 4) if num_phrases else 0
|
||||
avg_ent_words = round(sum_phrase_words / num_phrases, 4) if num_phrases else 0
|
||||
return OpenIE(
|
||||
docs=all_docs,
|
||||
avg_ent_chars=avg_ent_chars,
|
||||
avg_ent_words=avg_ent_words,
|
||||
)
|
||||
|
||||
def _to_dict(self):
|
||||
"""转换为字典"""
|
||||
return {
|
||||
"docs": self.docs,
|
||||
"avg_ent_chars": self.avg_ent_chars,
|
||||
"avg_ent_words": self.avg_ent_words,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def load() -> "OpenIE":
|
||||
"""从OPENIE_DIR下所有json文件合并加载OpenIE数据"""
|
||||
openie_dir = os.path.join(DATA_PATH, "openie")
|
||||
if not os.path.exists(openie_dir):
|
||||
raise Exception(f"OpenIE数据目录不存在: {openie_dir}")
|
||||
json_files = sorted(glob.glob(os.path.join(openie_dir, "*.json")))
|
||||
data_list = []
|
||||
for file in json_files:
|
||||
with open(file, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
data_list.append(data)
|
||||
if not data_list:
|
||||
# print(f"111111111111111111111Root Path : \n{ROOT_PATH}")
|
||||
raise Exception(f"未在 {openie_dir} 找到任何OpenIE json文件")
|
||||
openie_data = OpenIE._from_dict(data_list)
|
||||
return openie_data
|
||||
|
||||
def extract_entity_dict(self):
|
||||
"""提取实体列表"""
|
||||
ner_output_dict = dict(
|
||||
{
|
||||
doc_item["idx"]: doc_item["extracted_entities"]
|
||||
for doc_item in self.docs
|
||||
if len(doc_item["extracted_entities"]) > 0
|
||||
}
|
||||
)
|
||||
return ner_output_dict
|
||||
|
||||
def extract_triple_dict(self):
|
||||
"""提取三元组列表"""
|
||||
triple_output_dict = dict(
|
||||
{
|
||||
doc_item["idx"]: doc_item["extracted_triples"]
|
||||
for doc_item in self.docs
|
||||
if len(doc_item["extracted_triples"]) > 0
|
||||
}
|
||||
)
|
||||
return triple_output_dict
|
||||
|
||||
def extract_raw_paragraph_dict(self):
|
||||
"""提取原始段落"""
|
||||
raw_paragraph_dict = dict({doc_item["idx"]: doc_item["passage"] for doc_item in self.docs})
|
||||
return raw_paragraph_dict
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 测试代码
|
||||
print(ROOT_PATH)
|
||||
70
src/chat/knowledge/prompt_template.py
Normal file
70
src/chat/knowledge/prompt_template.py
Normal file
@@ -0,0 +1,70 @@
|
||||
entity_extract_system_prompt = """你是一个性能优异的实体提取系统。请从段落中提取出所有实体，并以JSON列表的形式输出。
|
||||
|
||||
输出格式示例：
|
||||
[ "实体A", "实体B", "实体C" ]
|
||||
|
||||
请注意以下要求：
|
||||
- 将代词（如“你”、“我”、“他”、“她”、“它”等）转化为对应的实体命名，以避免指代不清。
|
||||
- 尽可能多的提取出段落中的全部实体；
|
||||
"""
|
||||
|
||||
|
||||
def build_entity_extract_context(paragraph: str) -> str:
|
||||
"""构建实体提取的完整提示文本"""
|
||||
return f"""{entity_extract_system_prompt}
|
||||
|
||||
段落：
|
||||
```
|
||||
{paragraph}
|
||||
```"""
|
||||
|
||||
|
||||
rdf_triple_extract_system_prompt = """你是一个性能优异的RDF（资源描述框架，由节点和边组成，节点表示实体/资源、属性，边则表示了实体和实体之间的关系以及实体和属性的关系。）构造系统。你的任务是根据给定的段落和实体列表构建RDF图。
|
||||
|
||||
请使用JSON回复，使用三元组的JSON列表输出RDF图中的关系（每个三元组代表一个关系）。
|
||||
|
||||
输出格式示例：
|
||||
[
|
||||
["某实体","关系","某属性"],
|
||||
["某实体","关系","某实体"],
|
||||
["某资源","关系","某属性"]
|
||||
]
|
||||
|
||||
请注意以下要求：
|
||||
- 每个三元组应包含每个段落的实体命名列表中的至少一个命名实体，但最好是两个。
|
||||
- 将代词（如“你”、“我”、“他”、“她”、“它”等）转化为对应的实体命名，以避免指代不清。
|
||||
"""
|
||||
|
||||
|
||||
def build_rdf_triple_extract_context(paragraph: str, entities: str) -> str:
|
||||
"""构建RDF三元组提取的完整提示文本"""
|
||||
return f"""{rdf_triple_extract_system_prompt}
|
||||
|
||||
段落：
|
||||
```
|
||||
{paragraph}
|
||||
```
|
||||
|
||||
实体列表：
|
||||
```
|
||||
{entities}
|
||||
```"""
|
||||
|
||||
|
||||
qa_system_prompt = """
|
||||
你是一个性能优异的QA系统。请根据给定的问题和一些可能对你有帮助的信息作出回答。
|
||||
|
||||
请注意以下要求：
|
||||
- 你可以使用给定的信息来回答问题，但请不要直接引用它们。
|
||||
- 你的回答应该简洁明了，避免冗长的解释。
|
||||
- 如果你无法回答问题，请直接说“我不知道”。
|
||||
"""
|
||||
|
||||
|
||||
# def build_qa_context(question: str, knowledge: list[tuple[str, str, str]]) -> list[LLMMessage]:
|
||||
# knowledge = "\n".join([f"{i + 1}. 相关性：{k[0]}\n{k[1]}" for i, k in enumerate(knowledge)])
|
||||
# messages = [
|
||||
# LLMMessage("system", qa_system_prompt).to_dict(),
|
||||
# LLMMessage("user", f"问题：\n{question}\n\n可能有帮助的信息：\n{knowledge}").to_dict(),
|
||||
# ]
|
||||
# return messages
|
||||
124
src/chat/knowledge/qa_manager.py
Normal file
124
src/chat/knowledge/qa_manager.py
Normal file
@@ -0,0 +1,124 @@
|
||||
import time
|
||||
from typing import Tuple, List, Dict, Optional
|
||||
|
||||
from .global_logger import logger
|
||||
from .embedding_store import EmbeddingManager
|
||||
from .kg_manager import KGManager
|
||||
|
||||
# from .lpmmconfig import global_config
|
||||
from .utils.dyn_topk import dyn_select_top_k
|
||||
from src.llm_models.utils_model import LLMRequest
|
||||
from src.chat.utils.utils import get_embedding
|
||||
from src.config.config import global_config, model_config
|
||||
|
||||
MAX_KNOWLEDGE_LENGTH = 10000 # 最大知识长度
|
||||
|
||||
|
||||
class QAManager:
|
||||
def __init__(
|
||||
self,
|
||||
embed_manager: EmbeddingManager,
|
||||
kg_manager: KGManager,
|
||||
):
|
||||
self.embed_manager = embed_manager
|
||||
self.kg_manager = kg_manager
|
||||
self.qa_model = LLMRequest(model_set=model_config.model_task_config.lpmm_qa, request_type="lpmm.qa")
|
||||
|
||||
async def process_query(self, question: str) -> Optional[Tuple[List[Tuple[str, float, float]], Optional[Dict[str, float]]]]:
|
||||
"""处理查询"""
|
||||
|
||||
# 生成问题的Embedding
|
||||
part_start_time = time.perf_counter()
|
||||
question_embedding = await get_embedding(question)
|
||||
if question_embedding is None:
|
||||
logger.error("生成问题Embedding失败")
|
||||
return None
|
||||
part_end_time = time.perf_counter()
|
||||
logger.debug(f"Embedding用时：{part_end_time - part_start_time:.5f}s")
|
||||
|
||||
# 根据问题Embedding查询Relation Embedding库
|
||||
part_start_time = time.perf_counter()
|
||||
relation_search_res = self.embed_manager.relation_embedding_store.search_top_k(
|
||||
question_embedding,
|
||||
global_config.lpmm_knowledge.qa_relation_search_top_k,
|
||||
)
|
||||
if relation_search_res is None:
|
||||
return None
|
||||
# 过滤阈值
|
||||
# 考虑动态阈值：当存在显著数值差异的结果时，保留显著结果；否则，保留所有结果
|
||||
relation_search_res = dyn_select_top_k(relation_search_res, 0.5, 1.0)
|
||||
if not relation_search_res or relation_search_res[0][1] < global_config.lpmm_knowledge.qa_relation_threshold:
|
||||
# 未找到相关关系
|
||||
logger.debug("未找到相关关系，跳过关系检索")
|
||||
relation_search_res = []
|
||||
|
||||
part_end_time = time.perf_counter()
|
||||
logger.debug(f"关系检索用时：{part_end_time - part_start_time:.5f}s")
|
||||
|
||||
for res in relation_search_res:
|
||||
rel_str = self.embed_manager.relation_embedding_store.store.get(res[0]).str
|
||||
print(f"找到相关关系，相似度：{(res[1] * 100):.2f}% - {rel_str}")
|
||||
|
||||
# TODO: 使用LLM过滤三元组结果
|
||||
# logger.info(f"LLM过滤三元组用时：{time.time() - part_start_time:.2f}s")
|
||||
# part_start_time = time.time()
|
||||
|
||||
# 根据问题Embedding查询Paragraph Embedding库
|
||||
part_start_time = time.perf_counter()
|
||||
paragraph_search_res = self.embed_manager.paragraphs_embedding_store.search_top_k(
|
||||
question_embedding,
|
||||
global_config.lpmm_knowledge.qa_paragraph_search_top_k,
|
||||
)
|
||||
part_end_time = time.perf_counter()
|
||||
logger.debug(f"文段检索用时：{part_end_time - part_start_time:.5f}s")
|
||||
|
||||
if len(relation_search_res) != 0:
|
||||
logger.info("找到相关关系，将使用RAG进行检索")
|
||||
# 使用KG检索
|
||||
part_start_time = time.perf_counter()
|
||||
result, ppr_node_weights = self.kg_manager.kg_search(
|
||||
relation_search_res, paragraph_search_res, self.embed_manager
|
||||
)
|
||||
part_end_time = time.perf_counter()
|
||||
logger.info(f"RAG检索用时：{part_end_time - part_start_time:.5f}s")
|
||||
else:
|
||||
logger.info("未找到相关关系，将使用文段检索结果")
|
||||
result = paragraph_search_res
|
||||
ppr_node_weights = None
|
||||
|
||||
# 过滤阈值
|
||||
result = dyn_select_top_k(result, 0.5, 1.0)
|
||||
|
||||
for res in result:
|
||||
raw_paragraph = self.embed_manager.paragraphs_embedding_store.store[res[0]].str
|
||||
print(f"找到相关文段，相关系数：{res[1]:.8f}\n{raw_paragraph}\n\n")
|
||||
|
||||
return result, ppr_node_weights
|
||||
|
||||
async def get_knowledge(self, question: str) -> Optional[str]:
|
||||
"""获取知识"""
|
||||
# 处理查询
|
||||
processed_result = await self.process_query(question)
|
||||
if processed_result is not None:
|
||||
query_res = processed_result[0]
|
||||
# 检查查询结果是否为空
|
||||
if not query_res:
|
||||
logger.debug("知识库查询结果为空，可能是知识库中没有相关内容")
|
||||
return None
|
||||
|
||||
knowledge = [
|
||||
(
|
||||
self.embed_manager.paragraphs_embedding_store.store[res[0]].str,
|
||||
res[1],
|
||||
)
|
||||
for res in query_res
|
||||
]
|
||||
found_knowledge = "\n".join(
|
||||
[f"第{i + 1}条知识：{k[0]}\n 该条知识对于问题的相关性：{k[1]}" for i, k in enumerate(knowledge)]
|
||||
)
|
||||
if len(found_knowledge) > MAX_KNOWLEDGE_LENGTH:
|
||||
found_knowledge = found_knowledge[:MAX_KNOWLEDGE_LENGTH] + "\n"
|
||||
return found_knowledge
|
||||
else:
|
||||
logger.debug("LPMM知识库并未初始化，可能是从未导入过知识...")
|
||||
return None
|
||||
0
src/chat/knowledge/utils/__init__.py
Normal file
0
src/chat/knowledge/utils/__init__.py
Normal file
51
src/chat/knowledge/utils/dyn_topk.py
Normal file
51
src/chat/knowledge/utils/dyn_topk.py
Normal file
@@ -0,0 +1,51 @@
|
||||
from typing import List, Any, Tuple
|
||||
|
||||
|
||||
def dyn_select_top_k(
|
||||
score: List[Tuple[Any, float]], jmp_factor: float, var_factor: float
|
||||
) -> List[Tuple[Any, float, float]]:
|
||||
"""动态TopK选择"""
|
||||
# 检查输入列表是否为空
|
||||
if not score:
|
||||
return []
|
||||
|
||||
# 按照分数排序（降序）
|
||||
sorted_score = sorted(score, key=lambda x: x[1], reverse=True)
|
||||
|
||||
# 归一化
|
||||
max_score = sorted_score[0][1]
|
||||
min_score = sorted_score[-1][1]
|
||||
normalized_score = []
|
||||
for score_item in sorted_score:
|
||||
normalized_score.append(
|
||||
tuple(
|
||||
[
|
||||
score_item[0],
|
||||
score_item[1],
|
||||
(score_item[1] - min_score) / (max_score - min_score),
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
# 寻找跳变点：score变化最大的位置
|
||||
jump_idx = 0
|
||||
for i in range(1, len(normalized_score)):
|
||||
if abs(normalized_score[i][2] - normalized_score[i - 1][2]) > abs(
|
||||
normalized_score[jump_idx][2] - normalized_score[jump_idx - 1][2]
|
||||
):
|
||||
jump_idx = i
|
||||
# 跳变阈值
|
||||
jump_threshold = normalized_score[jump_idx][2]
|
||||
|
||||
# 计算均值
|
||||
mean_score = sum([s[2] for s in normalized_score]) / len(normalized_score)
|
||||
# 计算方差
|
||||
var_score = sum([(s[2] - mean_score) ** 2 for s in normalized_score]) / len(normalized_score)
|
||||
|
||||
# 动态阈值
|
||||
threshold = jmp_factor * jump_threshold + (1 - jmp_factor) * (mean_score + var_factor * var_score)
|
||||
|
||||
# 重新过滤
|
||||
res = [s for s in normalized_score if s[2] > threshold]
|
||||
|
||||
return res
|
||||
8
src/chat/knowledge/utils/hash.py
Normal file
8
src/chat/knowledge/utils/hash.py
Normal file
@@ -0,0 +1,8 @@
|
||||
import hashlib
|
||||
|
||||
|
||||
def get_sha256(string: str) -> str:
|
||||
"""获取字符串的SHA256值"""
|
||||
sha256 = hashlib.sha256()
|
||||
sha256.update(string.encode("utf-8"))
|
||||
return sha256.hexdigest()
|
||||
98
src/chat/knowledge/utils/json_fix.py
Normal file
98
src/chat/knowledge/utils/json_fix.py
Normal file
@@ -0,0 +1,98 @@
|
||||
import json
|
||||
from json_repair import repair_json
|
||||
|
||||
|
||||
def _find_unclosed(json_str):
|
||||
"""
|
||||
Identifies the unclosed braces and brackets in the JSON string.
|
||||
|
||||
Args:
|
||||
json_str (str): The JSON string to analyze.
|
||||
|
||||
Returns:
|
||||
list: A list of unclosed elements in the order they were opened.
|
||||
"""
|
||||
unclosed = []
|
||||
inside_string = False
|
||||
escape_next = False
|
||||
|
||||
for char in json_str:
|
||||
if inside_string:
|
||||
if escape_next:
|
||||
escape_next = False
|
||||
elif char == "\\":
|
||||
escape_next = True
|
||||
elif char == '"':
|
||||
inside_string = False
|
||||
else:
|
||||
if char == '"':
|
||||
inside_string = True
|
||||
elif char in "{[":
|
||||
unclosed.append(char)
|
||||
elif char in "}]":
|
||||
if unclosed and ((char == "}" and unclosed[-1] == "{") or (char == "]" and unclosed[-1] == "[")):
|
||||
unclosed.pop()
|
||||
|
||||
return unclosed
|
||||
|
||||
|
||||
# The following code is used to fix a broken JSON string.
|
||||
# From HippoRAG2 (GitHub: OSU-NLP-Group/HippoRAG)
|
||||
def fix_broken_generated_json(json_str: str) -> str:
|
||||
"""
|
||||
Fixes a malformed JSON string by:
|
||||
- Removing the last comma and any trailing content.
|
||||
- Iterating over the JSON string once to determine and fix unclosed braces or brackets.
|
||||
- Ensuring braces and brackets inside string literals are not considered.
|
||||
|
||||
If the original json_str string can be successfully loaded by json.loads(), will directly return it without any modification.
|
||||
|
||||
Args:
|
||||
json_str (str): The malformed JSON string to be fixed.
|
||||
|
||||
Returns:
|
||||
str: The corrected JSON string.
|
||||
"""
|
||||
|
||||
try:
|
||||
# Try to load the JSON to see if it is valid
|
||||
json.loads(json_str)
|
||||
return json_str # Return as-is if valid
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# Step 1: Remove trailing content after the last comma.
|
||||
last_comma_index = json_str.rfind(",")
|
||||
if last_comma_index != -1:
|
||||
json_str = json_str[:last_comma_index]
|
||||
|
||||
# Step 2: Identify unclosed braces and brackets.
|
||||
unclosed_elements = _find_unclosed(json_str)
|
||||
|
||||
# Step 3: Append the necessary closing elements in reverse order of opening.
|
||||
closing_map = {"{": "}", "[": "]"}
|
||||
for open_char in reversed(unclosed_elements):
|
||||
json_str += closing_map[open_char]
|
||||
|
||||
return json_str
|
||||
|
||||
|
||||
def new_fix_broken_generated_json(json_str: str) -> str:
|
||||
"""
|
||||
使用 json-repair 库修复格式错误的 JSON 字符串。
|
||||
|
||||
如果原始 json_str 字符串可以被 json.loads() 成功加载，则直接返回而不进行任何修改。
|
||||
|
||||
参数:
|
||||
json_str (str): 需要修复的格式错误的 JSON 字符串。
|
||||
|
||||
返回:
|
||||
str: 修复后的 JSON 字符串。
|
||||
"""
|
||||
try:
|
||||
# 尝试加载 JSON 以查看其是否有效
|
||||
json.loads(json_str)
|
||||
return json_str # 如果有效则按原样返回
|
||||
except json.JSONDecodeError:
|
||||
# 如果无效，则尝试修复它
|
||||
return repair_json(json_str)
|
||||
Reference in New Issue
Block a user