Sunday, March 25, 2012

Getting to the Big Data Problem

The amount of data in your organization is big. You just won’t believe how vastly, hugely, mind- bogglingly big it is. I mean you might think that it take a long time to read your inbox but that’s just peanuts to how much your organization touches in a single day.
- Mangling of quote by Douglas Adams in Hitchhiker’s Guide to the Galaxy
The amount of data in your organization is massive. Anyone who has been in the Content Management industry for more than a few years can tell you that much. All those content repositories are nothing more than messy, poorly structured, data warehouses.
The part that I didn’t realize until watching Clay Shirky’s keynote at AIIM 2012 was that the amount of data that many organizations is amassing isn’t always enough. Many organizations just are dealing with what I will now shockingly classify as “traditional” Big Data issue. They don’t have the volume, variability, variety, or velocity of data. (Your actual “V”s may vary)
This is actually an even bigger problem. All those organizations are leaving useful insights on the table. How can tell what my customers, employees , or members need without actually tracking what they are doing?
imageLet’s talk about you, the reader. If you are reading this, you are likely an Information Professional and the kind of person that AIIM wants to help become better at solving all these information challenges.

Sunday, March 4, 2012

e-CFR Data for electronic records management

e-CFR Data is current as of March 1, 2012 Return to search results Amendment from October 02, 2009 PART 1236—ELECTRONIC RECORDS MANAGEMENT Section Contents § 1236.1 What are the authorities for part 1236? § 1236.2 What definitions apply to this part? § 1236.4 What standards are used as guidance for this part? § 1236.6 What are agency responsibilities for electronic records management? § 1236.10 What records management controls must agencies establish for records in electronic information systems? § 1236.12 What other records management and preservation considerations must be incorporated into the design, development, and implementation of electronic information systems? § 1236.14 What must agencies do to protect records against technological obsolescence? § 1236.20 What are appropriate recordkeeping systems for electronic records? § 1236.22 What are the additional requirements for managing electronic mail records? § 1236.24 What are the additional requirements for managing unstructured electronic records? § 1236.26 What actions must agencies take to maintain electronic information systems? § 1236.28 What additional requirements apply to the selection and maintenance of electronic records storage media for permanent records? Subpart A—GeneralSec.1236.1What are the authorities for part 1236?1236.2What definitions apply to this part?1236.4What standards are used as guidance for this part?1236.6What are agency responsibilities for electronic records management?Subpart B—Records Management and Preservation Considerations for Designing and Implementing Electronic Information Systems1236.10What records management controls must agencies establish for records in electronic information systems?1236.12What other records management and preservation considerations must be incorporated into the design, development, and implementation of electronic information systems?1236.14What must agencies do to protect records against technological obsolescence?Subpart C—Additional Requirements for Electronic Records1236.20What are appropriate recordkeeping systems for electronic records?1236.22What are the additional requirements for managing electronic mail records?1236.24What are the additional requirements for managing unstructured electronic records?1236.26What actions must agencies take to maintain electronic information systems?1236.28What additional requirements apply to the selection and maintenance of electronic records storage media for permanent records? Authority: 44 U.S.C. 2904, 3101, 3102, and 3105. Subpart A—General § 1236.1 What are the authorities for part 1236? top The statutory authority for this part is 44 U.S.C. 2904, 3101, 3102, and 3105. OMB Circular A–130, Management of Federal Information Resources, applies to records and information systems containing records. § 1236.2 What definitions apply to this part? top (a) See §1220.18 of this subchapter for definitions of terms used throughout Subchapter B, including part 1236. (b) As used in part 1236— Electronic information systemmeans an information system that contains and provides access to computerized Federal records and other information. Electronic mail systemmeans a computer application used to create, receive, and transmit messages and other documents. Excluded from this definition are file transfer utilities (software that transmits files between users but does not retain any transmission data), data systems used to collect and process data that have been organized into data files or data bases on either personal computers or mainframe computers, and word processing documents not transmitted on an e-mail system. Metadataconsists of preserved contextual information describing the history, tracking, and/or management of an electronic document. Unstructured electronic recordsmeans records created using office automation applications such as electronic mail and other messaging applications, word processing, or presentation software. § 1236.4 What standards are used as guidance for this part? top These regulations conform with ISO 15489–1:2001. Paragraph 9.6 (Storage and handling) is relevant to this part. § 1236.6 What are agency responsibilities for electronic records management? top Agencies must: (a) Incorporate management of electronic records into the records management activities required by parts 1220–1235 of this subchapter; (b) Integrate records management and preservation considerations into the design, development, enhancement, and implementation of electronic information systems in accordance with subpart B of this part; and (c) Appropriately manage electronic records in accordance with subpart C of this part. Subpart B—Records Management and Preservation Considerations for Designing and Implementing Electronic Information Systems § 1236.10 What records management controls must agencies establish for records in electronic information systems? top The following types of records management controls are needed to ensure that Federal records in electronic information systems can provide adequate and proper documentation of agency business for as long as the information is needed. Agencies must incorporate controls into the electronic information system or integrate them into a recordkeeping system that is external to the information system itself (see §1236.20 of this part). (a)Reliability:Controls to ensure a full and accurate representation of the transactions, activities or facts to which they attest and can be depended upon in the course of subsequent transactions or activities. (b)Authenticity:Controls to protect against unauthorized addition, deletion, alteration, use, and concealment. (c)Integrity:Controls, such as audit trails, to ensure records are complete and unaltered. (d)Usability:Mechanisms to ensure records can be located, retrieved, presented, and interpreted. (e)Content:Mechanisms to preserve the information contained within the record itself that was produced by the creator of the record; (f)Context:Mechanisms to implement cross-references to related records that show the organizational, functional, and operational circumstances about the record, which will vary depending upon the business, legal, and regulatory requirements of the business activity; and (g) Structure: controls to ensure the maintenance of the physical and logical format of the records and the relationships between the data elements. § 1236.12 What other records management and preservation considerations must be incorporated into the design, development, and implementation of electronic information systems? top As part of the capital planning and systems development life cycle processes, agencies must ensure: (a) That records management controls (see §1236.10) are planned and implemented in the system; (b) That all records in the system will be retrievable and usable for as long as needed to conduct agency business (i.e., for their NARA-approved retention period). Where the records will need to be retained beyond the planned life of the system, agencies must plan and budget for the migration of records and their associated metadata to new storage media or formats in order to avoid loss due to media decay or technology obsolescence. (See §1236.14.) (c) The transfer of permanent records to NARA in accordance with part 1235 of this subchapter. (d) Provision of a standard interchange format (e.g., ASCII or XML) when needed to permit the exchange of electronic documents between offices using different software or operating systems. § 1236.14 What must agencies do to protect records against technological obsolescence? top Agencies must design and implement migration strategies to counteract hardware and software dependencies of electronic records whenever the records must be maintained and used beyond the life of the information system in which the records are originally created or captured. To successfully protect records against technological obsolescence, agencies must: (a) Determine if the NARA-approved retention period for the records will be longer than the life of the system where they are currently stored. If so, plan for the migration of the records to a new system before the current system is retired. (b) Carry out upgrades of hardware and software in such a way as to retain the functionality and integrity of the electronic records created in them. Retention of record functionality and integrity requires: (1) Retaining the records in a usable format until their authorized disposition date. Where migration includes conversion of records, ensure that the authorized disposition of the records can be implemented after conversion; (2) Any necessary conversion of storage media to provide compatibility with current hardware and software; and (3) Maintaining a link between records and their metadata through conversion or migration, including capture of all relevant associated metadata at the point of migration (for both the records and the migration process). (c) Ensure that migration strategies address non-active electronic records that are stored off-line. Subpart C—Additional Requirements for Electronic Records § 1236.20 What are appropriate recordkeeping systems for electronic records? top (a)General. Agencies must use electronic or paper recordkeeping systems or a combination of those systems, depending on their business needs, for managing their records. Transitory e-mail may be managed as specified in §1236.22(c). (b)Electronic recordkeeping. Recordkeeping functionality may be built into the electronic information system or records can be transferred to an electronic recordkeeping repository, such as a DoD–5015.2 STD-certified product. The following functionalities are necessary for electronic recordkeeping: (1)Declare records. Assign unique identifiers to records. (2)Capture records. Import records from other sources, manually enter records into the system, or link records to other systems. (3)Organize records. Associate with an approved records schedule and disposition instruction. (4)Maintain records security. Prevent the unauthorized access, modification, or deletion of declared records, and ensure that appropriate audit trails are in place to track use of the records. (5)Manage access and retrieval. Establish the appropriate rights for users to access the records and facilitate the search and retrieval of records. (6)Preserve records. Ensure that all records in the system are retrievable and usable for as long as needed to conduct agency business and to meet NARA-approved dispositions. Agencies must develop procedures to enable the migration of records and their associated metadata to new storage media or formats in order to avoid loss due to media decay or technology obsolescence. (7)Execute disposition.Identify and effect the transfer of permanent records to NARA based on approved records schedules. Identify and delete temporary records that are eligible for disposal. Apply records hold or freeze on disposition when required. (c)Backup systems.System and file backup processes and media do not provide the appropriate recordkeeping functionalities and must not be used as the agency electronic recordkeeping system. § 1236.22 What are the additional requirements for managing electronic mail records? top (a) Agencies must issue instructions to staff on the following retention and management requirements for electronic mail records: (1) The names of sender and all addressee(s) and date the message was sent must be preserved for each electronic mail record in order for the context of the message to be understood. The agency may determine that other metadata is needed to meet agency business needs,e.g.,receipt information. (2) Attachments to electronic mail messages that are an integral part of the record must be preserved as part of the electronic mail record or linked to the electronic mail record with other related records. (3) If the electronic mail system identifies users by codes or nicknames or identifies addressees only by the name of a distribution list, retain the intelligent or full names on directories or distributions lists to ensure identification of the sender and addressee(s) of messages that are records. (4) Some e-mail systems provide calendars and task lists for users. These may meet the definition of Federal record. Calendars that meet the definition of Federal records are to be managed in accordance with the provisions of GRS 23, Item 5. (5) Draft documents that are circulated on electronic mail systems may be records if they meet the criteria specified in 36 CFR 1222.10(b) of this subchapter. (b) Agencies that allow employees to send and receive official electronic mail messages using a system not operated by the agency must ensure that Federal records sent or received on such systems are preserved in the appropriate agency recordkeeping system. (c) Agencies may elect to manage electronic mail records with very short-term NARA-approved retention periods (transitory records with a very short-term retention period of 180 days or less as provided by GRS 23, Item 7, or by a NARA-approved agency records schedule) on the electronic mail system itself, without the need to copy the record to a paper or electronic recordkeeping system, provided that: (1) Users do not delete the messages before the expiration of the NARA-approved retention period, and (2) The system's automatic deletion rules ensure preservation of the records until the expiration of the NARA-approved retention period. (d) Except for those electronic mail records within the scope of paragraph (c) of this section: (1) Agencies must not use an electronic mail system to store the recordkeeping copy of electronic mail messages identified as Federal records unless that system has all of the features specified in §1236.20(b) of this part. (2) If the electronic mail system is not designed to be a recordkeeping system, agencies must instruct staff on how to copy Federal records from the electronic mail system to a recordkeeping system. (e) Agencies that retain permanent electronic mail records scheduled for transfer to the National Archives must either store them in a format and on a medium that conforms to the requirements concerning transfer at 36 CFR part 1235 or maintain the ability to convert the records to the required format and medium at the time transfer is scheduled. (f) Agencies that maintain paper recordkeeping systems must print and file their electronic mail records with the related transmission and receipt data specified by the agency's electronic mail instructions. § 1236.24 What are the additional requirements for managing unstructured electronic records? top (a) Agencies that manage unstructured electronic records electronically must ensure that the records are filed in a recordkeeping system that meets the requirements in §1236.10, except that transitory e-mail may be managed in accordance with §1236.22(c). (b) Agencies that maintain paper files as their recordkeeping systems must establish policies and issue instructions to staff to ensure that unstructured records are printed out for filing in a way that captures any pertinent hidden text (such as comment fields) or structural relationships (e.g., among worksheets in spreadsheets or other complex documents) required to meet agency business needs. § 1236.26 What actions must agencies take to maintain electronic information systems? top (a) Agencies must maintain inventories of electronic information systems and review the systems periodically for conformance to established agency procedures, standards, and policies as part of the periodic reviews required by 44 U.S.C. 3506. The review should determine if the records have been properly identified and described, and if the schedule descriptions and retention periods reflect the current informational content and use. If not, agencies must submit an SF 115, Request for Records Disposition Authority, to NARA. (b) Agencies must maintain up-to-date documentation about electronic information systems that is adequate to: (1) Specify all technical characteristics necessary for reading and processing the records contained in the system; (2) Identify all inputs and outputs; (3) Define the contents of the files and records; (4) Determine restrictions on access and use; (5) Understand the purpose(s) and function(s) of the system; (6) Describe update cycles or conditions and rules for adding, changing, or deleting information in the system; and (7) Ensure the timely, authorized disposition of the records. § 1236.28 What additional requirements apply to the selection and maintenance of electronic records storage media for permanent records? top (a) Agencies must maintain the storage and test areas for electronic records storage media containing permanent and unscheduled records within the following temperature and relative humidity ranges: (1) Temperature—62° to 68 °F. (2) Relative humidity—35% to 45%. (b) Electronic media storage libraries and test or evaluation areas that contain permanent or unscheduled records must be smoke-free. (c) For additional guidance on the maintenance and storage of CDs and DVDS, agencies may consult the National Institute of Standards and Technology (NIST) Special Publication 500–252, Care and Handling of CDs and DVDs athttp://www.itl.nist.gov/iad/894.05/papers/CDandDVDCareandHandlingGuide.pdf,contact phone number (301) 975–6478. (d) Agencies must test magnetic computer tape media no more than 6 months prior to using them to store electronic records that are unscheduled or scheduled for permanent retention. This test should verify that the magnetic computer tape media are free of permanent errors and in compliance with NIST or industry standards. (e) Agencies must annually read a statistical sample of all magnetic computer tape media containing permanent and unscheduled records to identify any loss of data and to discover and correct the causes of data loss. In magnetic computer tape libraries with 1800 or fewer tape media, a 20% sample or a sample size of 50 media, whichever is larger, should be read. In magnetic computer tape libraries with more than 1800 media, a sample of 384 media should be read. Magnetic computer tape media with 10 or more errors should be replaced and, when possible, lost data must be restored. All other magnetic computer tape media which might have been affected by the same cause (i.e., poor quality tape, high usage, poor environment, improper handling) must be read and corrected as appropriate. (f) Before the media are 10 years old, agencies must copy permanent or unscheduled data on magnetic records storage media onto tested and verified new electronic media.

Saturday, March 3, 2012

Update from the GRS Team

Update from the GRS Team:

The General Records Schedules (GRS) Team has received the following comment on the recent RM Communication, AC 09.2012: Announcing a New General Records Schedule Team, and wanted to share their response in the event that the expressed concern is shared by others.


Comment: It is just our experienced opinion that before the GRS Team embarks on the “much needed” Schedule for Web and Social Media Records, NARA might consider filling the glaring gaps for records that have been missing in the Schedules for years and really are “much needed” (e.g. Environment, Health, and Safety, Annual Employee Engagement Survey, and the new series created as a result of shared automated systems for Federal Travel, Employee Benefits, Payroll that are not clearly addressed, just a few examples).  Then, NARA can consider Social Media.  NARA has no idea how to manage Social Media records (mounted on third party sites) at this early stage, so clear guidance, as opposed to a GRS is needed, rather than preliminary generalized scheduling.


GRS Team response: The GRS Team does plan on addressing gaps in the GRS.  However, before we can address those gaps we need to identify what they are.  Part of our upcoming survey (see AC 10.2012)  asks agencies to let us know what is missing so that we know what we need to add.  Many of the gaps specifically noted in your e-mail relate to existing GRS schedules and those records will be addressed as we go through the revision process.  Those that are completely new categories will be worked into our overall plan as well.


There has been a call from Federal agencies for NARA to work on a new GRS for web and social media records.  It was one of the main concerns addressed to the Chief Records Officer at the recent ARMA meeting.  We chose it as the first new GRS project because NARA and the Federal Records Council have already been working towards a GRS to cover social media records and because it was a readily identifiable need.  We recognize that further guidance is needed in regard to managing web and social media records and we believe that drafting a GRS is a step forward.


We welcome further comments and input as we proceed with updating the GRS.  As stated above we will be sending out a survey to Federal Records Officers to gather some basic input, followed by some web and hopefully in-person sessions to delve further into issues and address agency needs. If you would like to comment or express concerns about the GRS or the GRS revision process, please feel free to contact us at GRSTeam@nara.gov.  We welcome your input in order to improve the GRS to better meet agency needs.




National Archives Frequently Asked Questions

Frequently Asked Questions

These are introductory answers to frequently asked questions about the National Archives and Records Administration (NARA) and our holdings and services.

Select a subject from the menu on the left to view questions and answers relating to your selection.
Links will guide you to further information on our web site or to other sources.

National Archives Frequently Asked Questions