A self-organizing system for automatic information classification and retrieval

In Cybernetics, Norbert Wiener presents his thoughts on self-organizing systems as they react and adapt to their environment. In his chapter on Information, Language, and Society, he points out the “well-known tendency of libraries to become clogged by their own volume” and notes the limitations of classification schemes, largely because of the problem of integrating new subject matter into a predetermined classification code. This paper introduces a self-organizing system that builds on Norbert Wiener's thoughts. It addresses the problem of information overload we are all currently experiencing in “big data” environments. Analyses of command information flows during military crises have suggested an approach to ontogenetic learning that avoids both the problem of describing the subject covered in a document and the problem of integrating new subject matter into a predetermined classification code. “Reference-connected sets” are constructed from message traffic dealing with a crisis and have been found to uniquely identify operational events during the crisis. Such sets (and subsets if further refinement is desired} can be obtained in real-time and a method is demonstrated that automatically classifies messages as they enter the system. The system thus focuses on the evolution of subjects as represented by referenced documents upon which any “new” information is based, in a manner similar to Norbert Wiener's “emergent behavior”. The paper then develops measures of effectiveness and shows how reference-connected sets can be used in filtering and organizing information in both hierarchical and non-hierarchical organizations across local, national, and international boundaries.