Version 4.10.2 Release

Summary

Version 4.10.2 includes a number of architectural changes, which continue the development of the open architecture and plug-in support. This includes, among other things, the following categories of changes:

• The builder includes many updates, but most importantly, generation of the HTML static files is mostly relegated to the web application.
• The system now supports result viewer plug-ins as well and a new timeline result viewer plug-in is currently under development.
• Potential customization of the digital resource pages was greatly increased with a new item layout configuration, allowing complete control of the page layout and support for a plug-in powered widget system on the page.
• HTTPS/SSL support was also increased, including a new worldwide CDN for HTTPS static content.

The full list of changes appears below.

Builder Updates

The builder application received many updates primarily focused at removing as most HTML creation from the builder, increasing the support for builder plug-ins, and additional search engine optimization features. In addition, several bug fixes and configuration updates were also included.

In previous versions of SobekCM, the builder would include the SobekCM HTML rendering library and would directly write HTML for canned browses, item views, and other outputs. In the new version, all the HTML empty frames and item level pages are pulled from the web application directly. The pages that require some HTML generation are added with an empty frame pulled from the web application. The new builder no longer links to the SobekCM_Library DLL, after moving several stored procedures between database gateway classes. This allows more of the overall look and feel of the generated pages to be controlled through the plug-ins in the web application and configuration.

The builder assists with overall search engine optimization by building static browse HTML files, RSS feeds, and static pages for each item to allow item metadata and files to be served quickly for search engine requests, without building the full item in memory. These item level static pages are now pulled from the /robot item viewer from the web application, but this can be changed via configuration or plug-ins. This new version includes a complete rewrite of the code that created the non-item pages for search engine indexing, which includes the sitemaps and rss HTML pages. In addition, sitemap creation was corrected for multi-instance builders and a new sitemap collection file is being created. On a daily basis, a schedulable builder module is called to recreate all the sitemaps and aggregation pages daily.

Additional code reorganization prepares to move some existing modules into plug-ins and increases ease when creating plug-ins. For example, the FDA ( Florida Dark Archive ) module was moved into its own code and the database calls moved into its own database gateway. Much of the base builder codes ( like the interfaces and abstract classes for the modules ) was moved out of the Builder library and into the Core code. This allows builder modules to be created that don't need to reference the Builder Library. This early requirement was making testing difficult due to circular redundancies in testing. Some final code reorganization moved much of the start-up functionality into schedulable builder modules which can be enabled or disabled in the database. The ability to schedule the tasks will be necessary when the builder is moved into a service running non-stop.

The following additional builder changes were also included:

• Changes to the main builder configuration files are immediately applied, especially used for multi-instance builders
• Corrected an issue with the deletion of materials through the bulk delete METS files
• Corrected the building of TIFFs from large JPEGs and prevented the builder from re-process items over and over if there is an error
• When checking JPEG2000 create time versus JPEG create time, a 2 minute buffer was added to stop JPEGs from being recreated unnecessarily

Results Viewer Configuration and Plug-Ins

A new architecture was created for the different result viewers, including a configuration file and reader and new tables in the database connecting aggregations with their result viewers. This included a code restructure of the existing result viewers and updates to the result viewer factory. In addition, the types of results views included in a collection, including the order and the default view, are now stored in the database.

Plug-ins for new result viewers are supported by the architecture, and a new timeline result viewer plug-in is currently under development.

Item Viewers and Item Layouts

In previous versions of SobekCM, the general look and feel of the item viewer was static. Over the last year, the item viewer plug-ins were introduced, which allowed a plug-in to alter the main viewer portion of the page when viewing an item. This new version builds upon those changes, adds the ability to control the overall look and feel within the item viewer in your instance by utilizing item layout configurations, and corrects a number of miscellaneous issues with the item viewers. A new architecture for inserting metadata into the HTML head on the digital resource pages was added allowing better search engine indexing while simultaneously increasing the ability to customize the metadata included.

Item layouts can now be utilized to alter the overall look and structure of the page that displays the individual digital resources. Layouts are defined within the configuration files and reference HTML layout files with all the stub HTML. Within the configuration and html file, sections are defined that can then have objects referenced in the configuration to have complete control over what item data is written in each section. The layouts can be chosen at the item level and individual item viewers can override the current item layout. This allows the quality control tool and page-turner, for example, to not utilize the left navigation bar at all.

Several updates support increased search engine optimization with the items. A new robots item viewer was created to provide the raw HTML for the builder to use when creating the static HTML source files for quick search robot requests. In addition, a new configurable architecture was added to allow the metadata written in the HTML head, and subsequently indexed by search engines, to be customized.

The web and builder no longer keep a complete list of every item in memory. For each (apparently valid) item request, a database check is performed. This also allows for better error to be provided. The errors include an invalid BibID requested and can even suggest an alternate VID if necessary. The main item writer now uses these codes from the MidTier engine as well.

The following additional item viewer changes were also included:

• Table of contents item viewer is now renders as HTML directly, without using ASP.net controls
• Corrected an issue where the Other URL metadata was missing in the default citation
• All methods in prototyper and viewer are now virtual, so they can be extended and selectively overriden
• Update for the change in the menu item adding function in the item viewer prototyper ( Add_Menu_items --> Add_Menu_Items )
• Downloads item viewer, including JPEG2000s, created, then moved into its own separate plugin (with Github repo)
• Fix to the GeoSpatial_BriefItemMapper for items with geo-spatial data only linked at the tile level
• Update fixes issue with the share buttons not working at the item level
• Corrected issue with the MODS reader which could get caught in the relatedItem tag if it was not "mods:relatedItem"

HTTPS (SSL) Support

Support for HTTPS/SSL access was updated and a new content delivery network is available for all the static files over HTTPS. Within the system-wide settings, the option to choose the static resource configuration source is now a special drop-down box, which lists all the existing static resource configurations from the config folders. A new HTTPS static resource configuration file is now included that points to https://cdn.sobekdigital.com for all the digital resources, which utilizes our corporate certificate. The world-wide content delivery network was enabled for that URL as well, allowing the content to be served from over 30 locations worldwide, for best performance.

Two additional minor updates were also included to finish the HTTPS/SSL support:

• JPEG2000 viewer SeaDragon URLs are now in the static content configuration file
• Corrected an issue that the draggable jquery UI link was not being read from the configuration file correctly

Item Management Updates

• Item submission / online editing templates
  • Aggregations can be added in the templates constants section and they now apply correctly
  • Code cleanup and changes to base elements within the templates
    • Changed names of classes and some properties to be correct capitalization per coding standard
    • Added ability to add a large number of events to each element as well (in the classes that extend them)
    • More work on the html events helper for the templates was completed
    • Converted all template elements to use XmlReader when reading inner configuration data from the template xml config readers, rather than XmlTextReader
• Removed default 'All rights reserved by the submitter' rights statement from new users
• Updated the VIDs that appear in the VID drop down when adding a new volume to a title to show the last added VID. (may need to add configuration for this if others do not like this)
• Updated stored procedure used for deleting items.. was missing a recently added table in the deletion process
• Corrected the project template to exclude viewers (was causing crash)

Miscell...

Version 4.10.1 Release

Version 4.10.1 includes many updates and fixes for issues introduced in the major architectural changes from version 4.10.0. This new version continues to make incremental changes to the architecture of configurations and plug-ins. Many small updates were implemented to the management interfaces as well. A new standard TEI plug-in was added which adds native support for TEI, including online editing and metadata extraction.

TEI Plug-In

• Added a new TEI plug-in with a number of TEI-related updates
  • Added the TEI item viewer and item viewer prototyper for displaying a TEI with a XSLT and CSS
  • Adds new TEI submittion mySobek viewer for submitting the new TEI item
  • Puts a 'Add new TEI Item' on the mySobek menu
  • TEI type items have the standard 'Edit Metadata' options changed to 'Edit TEI'
  • Users can either upload updated TEI or edit the TEI directly in a nice Ace editor screen
  • A new admin screen allows users to upload XSLTs, CSS, and mapping files to map from TEI metadata to METS/MODS/SobekCM
  • Also, added two standard ( TEI Boilerplate, and TEIC ) XSLTs to the TEI plug-in.
  • Added all the admin features to permission users to have access to different XSLT, CSS, and Mappings
  • Added a new top-level admin page to add permissions to multiple users at the same time
  • Added a new tab on the users screen to enabled TEI for that user and permission for single user
  • TEI is only enabled by using the TEI plug-in.. eventually all the TEI stuff will be in the plug-in, but currently some of it is in the base code

Builder Updates

• Item views
  • Updated stored procedure to save a single item to an existing group to now include the default views if not existing
  • List of item views now set to NULL before saving to the database
  • If the views list is NULL, the views are no longer marked excluded as soon as they are added
• Web derivative generations
  • Update to the way derivatives are created by the builder, allowing derivatives to be created from the JPEG2000 as well as PNG and GIF images.
  • JPEG attributes are always recalculated by the builder now.
  • Update to the create image derivatives to correct issue that METS was not pre-read
• Plug-Ins
  • Updated the builder to correctly work with plug-ins for builder modules
  • Builder configuration pulled from the engine endpoint now correctly includes the folder modules
  • Fix for builder when there are no plug-ins in some situation .. unexpected null fixed
  • Added sample Wolfsonian builder plug-in module
• Minor style update to incoming builder folders online system settings screen
• Solr indexes will be deleted by the builder if an item is dark or private
• Fix for builder when processing multiple instances and one or more are set to in-active
• Verified that new items loaded through the builder get all the default viewers correctly

Item Viewer Updates

• Fixed issue with page 9 reverting to non-zoomable when viewing an item and stepping through the pages
• Fixed the style on the show and hide TOC. Standard width in CSS was too narrow
• The share buttons at the item level work again. They were not getting the HTML snippets from the engine endpoints correctly.
• Breadcrumbs to the aggregations at the item level restored
• Fixed the downloads viewer to correctly use the built in file proxy for restricted materials ( SOBEK-256 )
• DARK and IP restricted issues
  • Corrected issue with the video player not correctly playing files that were IP restricted ( AURARIA-23 )
  • Restricted items viewers still suppressed, but not included on menu and RESTRICTED is included for IP restricted items
  • Citation is correctly suppressed for DARK if indicated in system-wide settings
• Citation viewer changes
  • Special citation section writers now obey the display label set in the citation config file
  • URLs are now utilized by the standard citation viewer correctly ( when no search code included for that field anyway )
  • Added the linked list of aggregations back to the citation screen of items
  • Citation now correctly includes a URL for any standard element, if one is given ( WOLF-50 )

Item Management Updates

• Corrected issue saving four or more icons from the item behavior saving procedure
• Item and the list of items within an title were not always being cleared from the cache correctly
• Previously uploaded files no longer appear in the upload files option. TIFF image resources were not being released correctly, preventing deletion.
• Quality Control tool
  • Corrected an issue deleting the first page from an item in QC when no divisions exist
  • QC tool had some incorrect code for pulling the SobekCM item, corrected that ( AURARIA-30 )
  • Corrected a couple issues with QC tool not allowing move and thumbnail selection correctly anymore
• Fixed the METS Validator object to better report a validation issues ( SOBEK-257 )
• Corrects issue in the item behaviors screen and submission forms if the user is not an internal user and no aggregations are provisioned for the user. ( USF )
• Corrected the equal check on the Identifier object to include identifier AND type, not just TYPE. Was leaving out a second identifier with a different type but the same ID.

Metadata Mapping Changes

• Corrects metadata configuration file that caused the VRACore sections to be ignored.
• New bibliographic mapping architecture
  • Adding ability to implement different bibliographic mappings in configuration and plugins
  • Allows the standard mapping to be replaced or augmented via plug-ins
  • Allows the mapping to be extended for additional metadata modules added through plug-ins
  • Previously, the mappable fields were filtered through an enumeration, which was not extensible.
  • Fix for metadata mapper object creation method where a null on code assembly threw an error
  • Added new sample bibliographic mapping object (Henderson_Bibliographic_Mapper) to show how that is done. Will likely convert into a plug-in.
  • Corrected serialization of the Metadata_Mapping_Config object (lacked serialization decorators at top class level previously)
  • Update to metadata mappings (for ContentDM import)
• MODS support changes
  • Added a new top level Table of Contents object, to allow display label to be used for the contents, and made it a list to support repeatability. (previously was a single string)
  • Added a new collection of other dates in the origin info tag. Other dates support type (can be used by display label) and key date as attributes.
• New generic XML mapper object
  • Added a new generic XML metadata reader, which reads data from a XML file into the SobekCM item object
  • Uses the new bibliographic mapping architecture and a XML configuration file that essentially maps between XPath and metadata field name
  • Utilized by the TEI plug-in

Aggregation Viewer Updates

• Corrected default SQL for collection browsing ( to remove Subject.Display and Publisher.Display from final display labels )
• Aggregation Management
  • Corrected default zoom of the custom map browse/search to zoom level 5, rather than zoom level 1, which equals a view of the whole globe
  • If aggregation html source was missing it was failing to rewrite after editing the home text if folder was missing
  • Added back missing ckeditor browser file which caused browsing images uploaded to server to fail when editing collection home pages
• Added a new tile-based home page for aggregations, which requires bib/vid names jpegs in the images/tiles subfolder under the aggregation design folder.
• Corrected an issue where referencing an unexisting aggregation would cause errors ( AURARIA-30 )
• Invalid collection subcode request
  • Received errors when NEW was requested for collections.
  • Corrected behavior when an invalid subcode is supplied for an aggregation.. now it defaults to home.

Web Skin Updates

• Headers / Footers
  • Added some exception catching around the writing of the header, for some debugging working on with USF.
  • Changed the way the headers/footers get the directives replaced to use StringBuilder for memory/performance and reformatted for ease of reading
  • Added the following new directives to be used in the header and footer
    • [%COLLCODE%] provides the collection code
    • [%BIBID%] provides the bibid
    • [%VID%] provides the vid
    • [%MODE%] provides a string for the current mode ( 'aggregation', 'results', or 'item' only currently )
  • The [%MYSOBEK%] directive now also works in the footer HTML in a web skin
• Web skin online management
  • Font files can now be uploaded in to the web skin through the web interface.
  • Editing the CSS for the web skin now uses the Ace Editor, giving syntax checks and syntax highlighting
  • Added ability to edit web-skin-specific javascript file in the online editing, and included in header as reference
  • Added second row of buttons at the bottom of the web skin management screen (can mess up header to make top row not accessible, I learned the hard way)

Search Results Changes

• Subtitle was added to the main title search term, allowing it to be searched with the other titles
• Facet column title ( Narrow Results By ) is no longer written as all-caps, but a text-transform was added to the sobekcm.css file. This can, of course, be overridden to NOT capitalize in your instance.
• New search system-wide setting flag indicates if the search explanation includes the ' results in XX records' ( i.e., your search for .. resulted in XX records ) or if that is omitted for searches with results.
• If a point in the map browse has only one item, clicking the link for more information about the results sends the user to the one item
• Corrected issue that the google map api was not dynamic in the map search results screen
• Corrected issue in the related item pop-up form HTML code

Plug-In Architecture Updates

• Corrected the item viewer factory to load item viewers fr...

Version 4.10.0

Summary

Version 4.10.0 of SobekCM includes the most comprehensive collection of changes since SobekCM was first released ten years ago. This version continues preparation for the release of version 5.0 by updating the basic architecture of the system to be more configurable and customizable. In addition, the web application is increasing dependent on the discrete microservice architecture provided by the SobekCM engine. This version also adds a new plug-in architecture for adding extensions which can add new metadata types, modify or add new item viewers, customize item citation display, add new engine endpoints, add new builder modules, and much more. Over thirty new microservice endpoints were also added to the SobekCM engine as well. The sum of all these changes results in a more open-architecture which can easily be modified and customized for any institution.

In addition, this version adds several new administrative screens for users. A new set of many system-wide settings screens exposes all configuration information used by the system currently, even if they cannot be modified directly through the web interface. In addition, interoperability between the SobekCM Builder service and the web has been increased by adding the ability to view and filter the builder log through the user interface. System administrators can also add or remove builder folders, directly through a new builder folder management screen.

The builder was also updated in this new version. The architecture was modified to take advantage of the new configuration microservices offered by the engine and supports enabled plug-ins as well. The automatic usage collection and monthly emails were updated to make the process run more smoothly and significantly quicker. The builder can also be set to use Tesseract to perform automatic OCR on incoming TIFF files.

In total, nearly one hundred thousand lines of code were changed between the previous version and this version, which represents roughly 35% of the total codebase and 66% percent of the files were changed in some way.

Architecture

The architecture of the SobekCM web application underwent major changes to provide a more open architecture based on microservice endpoints. The core of the system, the display of digital resources, was completely revamped and a new architecture for the item viewers was implemented. While metadata customization has existed in the METS reader and writer for some time, this version completes that process by allowing simple customization of the citation and online forms to allow additional metadata fields to be added fairly simple. In addition, far more portions of the system can be controlled via configuration files, to allow for limitless customization.

In general, the architecture is more focused on pulling the data from the SobekCM engine "just in time", instead of pulling all the data first, and then building the architecture to serve the response in the format requested. This reflects the change that the SobekCM web application will be depended on less to provide JSON and raw data for other tools, since the SobekCM engine endpoints now can provide this functionality through microservices. Collecting all of the data at the very beginning was far less extensible. With the new architecture, a completely different item viewer architecture could be created, using a different base object as well, and be swapped into the system for the current item HTML writer. Allowing changes like this necessitated a greater flexibility in object types and a decrease in the numebr of limiting enumerations utilized throughout the system.

Configuration

The overall ability to control the look and feel via configuration files was greatly enhanced in this version. During this process, five new default configuration files were added and all the separate configuration readers were consolidated into a single reader. All the sections of the configuration files are now repeatable and many levels of the hierarchy include an ability to clear all pre-existing configuration information. Previously, configuration information for each discrete portion of the system was controlled by a specific configuration file by filename. With this new release, the reader reads all the configuration files and any section of the configuration can be in any of the configuration files. All of the configuration information and the log file generated while reading all the configuration files are stored in the InstanceWide_Configuration object and are serializable and accessible through engine microservice endpoints as JSON, JSON-P, XML, and ProtoBuf.

Item Viewer Customization

This version includes major changes to the way item viewers work, although in most cases the resulting display will be very similar to the previous version. These changes normalized the behavior of all the existing item viewers while simultaneously allowing new viewers to be seamlessly added. In addition, the item writer utilizes the new BriefItemInfo object API and the new REST microservice endpoints for retrieving item information to display.

Each item viewer contains two classes, the item viewer prototype and the actual item viewer. The prototype is a singleton object that is used to determine if a particular viewer should be included for any digital resource. The prototype also determines if the user has access to the view and indicates how (or if) the viewer should be included on the item main menu. Finally, the item viewer prototype is used by the item view factory to create the item viewer itself which draws the HTML for a single view of a digital resource.

All the assembly and class information for each item viewer prototype is stored in the configuration files. Also in the configuration files is the URL segment that requests the viewer when someone is using the web interface and the code which matches back to the database viewer. The assembly and class to be used when writing the item main menu is included, which allows for even more customization of the main menu if desired. In the configuration information, individual viewers can be marked as management viewers, in which case they are always included in the items. Otherwise, all the views to be included in the resource are pulled from the database and included as a string, rather than a constricting enumeration. All of this makes it much easier to add new viewers to your own instance of SobekCM to fully customize the way the digital resource display works.

The database contains the basic priority of each viewer ( i.e., which should be the default viewer) as well as the order they appear on the menu. This can be changed system-wide if desired. In addition, it is now possible in the database to override this and set the viewer order and menu order for a single item or set of items.

The digital resource microservice APIs exposed by the engine, and the data format of the BriefItemInfo object was refined and should now be mostly stable moving forward. We are encouraging everyone to work with the item API if possible and adapt its use over any previously used APIs. The BriefItemInfo object was modified to include behavior information, geo-spatial data, hierarchical spatial subjects, and other data needed by the user interface. In addition, a structure was added to allow any item-level settings to be loaded from the database and an extension data structure was added to allow for extensibility of the BriefItemInfo object for any plug-ins or extensions.

Metadata Customization

Metadata creation was already very customizable, but several important changes were implemented in this version. A metadata reader/writer or METS section reader/writer can now be referenced from an external assembly in the configuration and the external assembly/class will be loaded. The existing metadata configuration objects were updated to allow configuration to be collected over many configuration files and it now exists in the SobekCM.Core library. The settings within the SobekCM.Resource_Object library can be set via the new static ResourceObjectSettings class. In addition, all the metadata configuration information is serializable so it can be served via the engine architecture.

Citation Customization

Display of the citation can now be customized via individual citation sets defined in the configuration. For each citation element, you can define the metadata term (mapping), display term, search code, Microdata ItemProp mapping, and several other behavior flags. Citation set can be specified for individual items through the item behavior screen, allowing the citation to be customized for a single item, or a set of items. The citation information included in emails sent through the system use their own citation set and can like-wise be customized.

For each citation element, you can either use the default citation writer, or you can implement your own citation section writer. Custom citation section writers exist, by default, for the Rights statement ( can include creative commons license image and link ), Coordinates ( based off the GeoSpatial object ), and Spatial Coverage ( hierarchical, with country, state, county, and city individually linked for searching ).

Template Element Customization

The mapping from the template files (which controls the user experience while working with online templates for metadata and behavior editing) was changed to allow for customization and addition of new template elements tied to new (or existing) metadata elements. A new configuration file was added which maps between the template type/subtype values and the individual template element objects to be utilized. The enumeration of types was removed to allow for the addition of new metadata elements. For inter-template element dependencies, a new system was developed where a dictionary is passed around to be written to and read by each template element in the template.

#...

Version 4.9.0

This version includes the addition of a robust web content system, useful for online exhibits, general announcements related to your digital collection, or documentation. This module is the largest single addition to the SobekCM system in the last five years, with more than 10 new administrative screens. In addition, all of the administrative and public pages are supported by the SobekCM Engine REST APIs. This makes this module the first to be supported 100% by the new architecture. As a result of this work, the architecture of the SobekCM Engine continued to evolve.
In addition to the web content work, several changes occurred to searching and browsing material, a new video viewer was added, and the system realize many small improvements. In addition, the efforts towards the cleanest code possible was continued and additional support for PostgreSQL and MySQL continued to be added. The ability to customize several messages, including the 404 missing page and no results pages was added. Finally, many community reported bugs have been corrected.
A complete list of changes appears below.

Web Content (Exhibits) Module

• Changed the way the URL is parsed to the web content pages
  • Web Content Hierarchy is now stored in the engine application state
  • Web Content Hierarchy is now checked for any possible calls before aggregation call checks
  • Only web content pages in the database will function now
• Added ability to set a redirect, so that a user hitting this page is redirected to another (internal or external) URL
• New single page administration
  • Enabled online editing of the HTML, including image upload, similar to the aggregation-level static HTML pages
  • Added new internal header for logged on, permissioned users
  • Added usage statistics viewer for one page online
  • Added milestone/history viewer to see all the changes through time
  • Permissions viewer to see assigned rights for page online (currently admin and portal admins)
  • Management menu to access all the new single page admins screens
  • Admin view:
    • Allows all of the SEO metadata to be edited
    • Manage the display characteristics for the page
    • Associate the page with a web content navigation unit
    • View all the child pages
    • Bulk manage uploaded images and documents
• New top level administration
  • View list of all pages and redirects
  • View usage statistics for all pages
  • View history of changes across all web content pages
  • Ability to add a new web content page
• New REST API endpoints are used for all of this, both public and admin access

Video Item Viewer

• Added new item viewer for MP4, OGG, and WEBM formats
• Automatically added when those files are available for download
• Includes ability to select the video to view if multiple

PostgreSQL and MySQL support

• Engine agnostic layer pulled into its own class library
• Resource object library database now uses this library for all database calls
• Removed reference to old Microsoft.ApplicationBlocks.Data library
• Fixed OAI-PMH saving error introduced with new support for PostgreSQL
• Support for selecting MySQL as database type

General REST API endpoint / SobekCM Engine updates

• General endpoints are now passed a debug flag, so debug eligibility can be defined at the microservice handler level
• Fix for the IP restriction check when there are more than 3 IP ranges included
• Engine config file and config reader updates
  • Config reader now reads BOTH the default and user config files and merges them
  • Now supports ClearAll and RemoveAll elements to clear what was previously loaded at any level
  • Documentation parts moved into an AutoDoc tag, and can now reside at the path level
  • IPs can be added in the user config to those standard ranges from the default config
• Multiple verbs ( POST / PUT / DELETE )
  • A single endpoint can now map each verb differently, for REST API endpoints
  • Tested and implemented this new architecture with the web content endpoints
• Error catching
  • Errors encountered by the client are now written to the trace route as well as re-thrown
  • If endpoint is not found, that is also written to the trace route

Searching Changes

• New basic search type
  • Includes a checkbox to include full text in search (USF, Auraria, SOAS)
  • Updated the admin screens to include this and make the choice of basic searches more obvious
• Seaching Fixes
  • Advanced search would return dark items in the search results list in certain situations - corrected stored procedure (UOC-7)
  • Corrected issue where simplified search used the wrong link ( i.e., advanced search where all terms hit the same field ) (USF/SOBEK-59)
  • Corrected issue where citation links were encoded incorrectly, which caused failure on diacritics (WOLF-35)
  • EXACT type searches (like when you click on a term in metadata browse) were not unicode compliant throughout (SOBEK-79)
• Verified that temporal searching and faceting is working correctly from publication date

Geographic Support

• Map Searching
  • Fixed map searching to correctly show the coordinates again, for all instances (AURARIA-21)
  • Corrected issue with two globe icons appearing in search results ( SOAS/SOBEK-32 )
  • Changed columns used when searching (see item level support below)
• Map Browse
  • Corrected issue that thumbnails were not appearing in map browse (AURARIA-21)
  • Fix to allow map browsing at the top, ALL collection level
• Item Level Support
  • Fixed issue showing the map view at the digital resource level without a search
  • Added new columns in database at item level for spatial footprint and distance, to replace the incorrectly named and populated Spatial_KML
  • When the create final citation stored procedure is called at the end of item save, the complete footprint is now calculated
• Administrative Settings
  • Can now set the default map in the map search and map browse, or set to extent
  • Modified procedure to get aggregation information from the database to include geographic extent

Changes to Missing Page and No Results Pages

• Missing Page
  • Customizable via a missing.html file at the top level of the web content folder for non-admins
  • Returns a 404 code for robots and search
  • For admins, a special screen is displayed that provides several possible actions:
    • Add a new web content page
    • Add a new aggregation alias
    • Add a new aggregation using the wizard
    • Add a new item
• No Results Page ( WOLF-15 / AURARIA-22 )
  • Customizable via a noresults.html file at the top level of the web content folder
  • Several special tokens in that file allow the same functionality as before

Reporting

• Added fix for item count statistics where title count was incorrect on the date range search (UF)
• Updated item count stats to allow users to change criteria and just get ALL items OR items with digital files, in addition to items that are milestone complete (SOAS-5)
• Corrected tab direction on the usage reports
• Corrected the styling on the private items report when more than one page of private titles
  • Buttons use correct styling
  • Number of titles displayed ( i.e., 1 - 20 of 45 matching titles ) now shown

Online Item Management

• Image (JPEG) Upload
  • Thumbnails generated from uploaded JPEGs now
  • Large JPEGs are saved, converted to TIFF, and then smaller versions made.
  • Builder will then generate the JPEG2000s
• When deleting a PDF in the online interface, the related text and thumbnail files are also deleted (SOBEK-52/USF)
• Updated Rights Element - UF copyright statements edits tements edits
• Updating the title on a single item Bib will also update the group title, unless it is a serial or newspaper type item
• Quality Control
  • Updated issue for recently uploaded images not receiving their proper name in the QC, causing an exception on final save
  • Updated QC exception catching to work correctly and to email the error in when it occurs (WOLF-29)
• Corrected issue deleting existing items (MBDA-14)

Builder Changes

• Added more logging to the usage statistics builder module
• Builder image derivative creation changes
  • Was not checking for JPEG2000 missing, only JPEGs. Now correctly checks JPEG2000s as well. (USF)
  • No longer copies the TIFF to a local temporary folder for processing. All processing happens in situ (SOBEK-51/81)
• Added more logging to the copy to archive builder module

Code Cleanup

• Documentation and Warnings
  • Update corrects hundreds of fairly inconsequential warnings due to incomplete XML documentation, unused variables, and other warnings. Only the SObekCM_Resource_Object is left. Also, ran the Resharper Code Cleanup routine against all the other libraries. Corrected many XML invalid comments.
  • Finished cleaning the web code and clearing nearly all the compilation (mostly documentation) warnings
  • Finished updating Builder_Library code for better documentation and cleaner, more conformant code with more comments and no warnings
• Removed Aware JPEG2000 viewer and query string stuff
• Removed an unnecessary call to tracking and history when showing the citation for an item for an item

Bug Fixes

• Fix for editing the home page text, now that child aggregations are included in the ALL
• Corrected issue where the host administrator check box was not saving to the database during user administration
• Corrected issue preventing user group changes from correctly saving (SOBEK-66/SOAS)
• Printing citation online now works correctly again (WOLF-34)
• Corrected HTML facet link for donors in the citation - ( donor ) was messing it up (WOLF-30)
• Corrected issue that DARK items were a little too resilient in loading downloads and images and were semi-accessible via the TOC (UOC-8)
• Verified that downloads with accents works just fine throughout the system (SOBEK-47)
• Check for null and check for existence of viewer behavior. Was causing a s...

Version 4.8.11

Overview

This update includes many small fixes and updates to the REST API. The REST API error catching and troubleshooting was generally improved upon and several new endpoints were added. The most robust added endpoint is the new search results endpoint. I expect some changes to occur to this API over for the next version though, in particular how items are displayed under the titles on search results and browses for deeply hierarchical material, like newspapers.

In addition, groundwork was laid for increased support for the top-level web content pages and for PostgreSQL and MySQL support as well.

General REST API endpoint updates

• New base class EndpointBase
  • Handles serialization based on protocol
  • Utilized new base class in all existing endpoint classes
• Better debugging and error catching to all endpoints
  • Added debug mode to nearly all existing endpoints ( debug=debug )
  • Added try/catch to nearly all existing endpoints and will show exception in debug mode
  • Added tracer object through the helped classes and added many trace events for debug mode
• Added new callback url query option on all JSON-P endpoints, to indicate the name of the callback method to use in the response
• Added children to ALL collection for REST API purposes, based on thematic headings
• Web skin REST API updates …
  • Web skin objects contain an exception string value
  • If the requested web skin does not exist (or the source files are not there) system more resilient
  • Caching of web skins improved
  • Debug trace routes for web skins improved
  • UI is now using the SobekEngineClient_WebSkinEndpoints class

New REST API Endpoints

• Search results
  • Added new class to hold the requested search arguments in the engine
  • Can do all searching or browsing within a collection
  • Added legacy API support for items (json) and results (json and xml) using new endpoints
  • Added main thumbnail to the legacy XML results view on new endpoint
• Top-level web content pages
• Dataset services

SobekCM engine client work

• Web skin engine client completed and is now used to get the web skins
• Web content engine client now supports getting the HTML based content and is now used
• Added the endpoint for adding a new web content page to the system

Search results changes

• Changed the way the standard facets are handled
• Now a list of Search_Facet_Collection objects
• Added serialization decorations to all the search results objects
• Modified the way facets are saved to the objects from the database search return values
• Decorated the Search_Facet_Aggregation class correctly for REST

General code reorganization

• Corrected the namespace of many of the brief item mapper classes
• Renamed 'SobekCM_Query_Analyzer' to just 'Query_Analyzer'
• DLL reorganization and cleanup
  • Removed several unused references in libraries
  • Fixed the csproj files where some of the DLLs were not referenced relative to solution folder

Top-level web content pages

• SQL Updates for increased top-level web content page CMS support
• Finished the database scripts, database gateway, and objects for new web content functionality

PostgreSQL support

• Update to add new database layer to abstract SQL engine
• Coded all the MS SQL calls
• Abstracted some concepts, like parameters, and examined Npgsql object hierarchy
• Updated all the database calls from the engine and the SobekCM_Library
• Refactored those database gateway classes for XML remarks and parameter naming
• Sunset the old Application Data library ( SqlHelper ) except for SobekCM_Resource_Object db
• Database in SobekCM_Resource_Object class needs to be updated - otherwise ready

OAI-PMH Updates

• OAI-PMH GetRecord response corrected
• OAI-PMH now uses a number of defaults for Identify, unless specified in config file
  • OAI-PMH Name = System Name
  • OAI-PMH Identifier = System Abbreviation
  • OAI-PMH Identifier base = oai:sysabbrev:
  • OAI-PMH email = System Emails
• Updated the default config file to specify very little of this data
• Default config file continues to specify oai_dc and marc21 as the default formats

Updates to the installer

• Slash direction for sql database / instance
• Correctly works for virtual directory installations (i.e., not at root level of a website)

Miscellaneous fixes

• Hover over for thumbnail view on search results has hover over restored
• Fix to editing the URL portals online.
• Fix a map search issue with pin not appearing on map correctly
• Fix an issue with coordinate searches that return no results showing divide by zero error
• Fix for issue with installation under a virtual directory may not record application network setting correctly
• Online submission and file management
  • Can now use [%SYSTEMNAME%] in the template agreement text
  • Added CSV to the default okay to upload files
  • Selecting the holding or source code sets the text automatically now
  • Holding and source metadata elements completed automatically now if only one option for user
• Added new flag and message to disable non-sys admins from logging on at all
• Corrected issue with PDF not going full screen in Chrome when a text search occurred

Version 4.8.10

Overview

This release increases the REST API support to include XML and also adds the url-resolver endpoint. The builder is updated to ensure all page images present are added to the items on reprocess and that all related viewers are added, as well as fixing an issue with the text searchable flag. This version also makes great strides in some final clean-up of the HTML5 and adds excellent accessibility support. Ability to customize the look and feel is incrementally increased again. A great many reported bugs are fixed and some UI enhancements included that were requested.

REST Endpoints

• Added a debugging option to the web skin endpoint, using the tracer
• Added new XML Endpoints
  • Language*specific item aggregation XML endpoint added
  • Brief and citation view of digital resource XML endpoints added
  • Language*specific web skin XML endpoint added
  • Added many XML decorators and enhancements for better generated XML
  • Changed many properties to nullable for cleaner responses
  • Fixed the many places in code that needed to do the null check now
  • Added new BriefItem_CitationResponse wrapper object for the 'citation' item response.
  • Tried and decided not to use the DataContractSerializer for SOAP output (configured XML is superior)
• URL-Resolver endpoint added
  • Endpoints for JSON, JSON*P, ProtoBuf, and XML added
  • Renamed SobekCM_Navigation_Object to Navigation_Object
  • Added many serialization decorators and enhancements to navigation object
  • Changed many properties to nullable for cleaner response
  • Added a new default NONE to all navigation enumerations
  • Removed internal user from the navigation object

Builder Changes

• Item Viewers
  • Builder adds thumbnail views and also the page turner if four or more jpegs
  • New system*wide setting controls if the page turner is added automatically or not
  • Corrected module so thumbnails for PDFs and PPTs and such aren't added as a page image (USF, UF)
  • Also, previously messed up packages are fixed by the process above (USF, UF)
• Text Searchable flag
  • TextSearchable flag is now set based on text files for page images (UOC-1)
  • Even existing items now have the TextSearchable flag set correctly (UOC-1)

Accessibility ( and better HTML5 )

• Added role="search" to all the basic searches (which appear on home pages) and advanced
• Updated HTML for search results in brief view
  • No longer uses the layout tables, everything is divs and definition lists
  • Hover over CSS change is now in CSS, not in javascript
  • Added label to the 'sort by' text related to the sort by select box
  • Added micro*data schema.org tag around the search results
• Updated sharing buttons to include onblur and onfocus, as well as onmouseout and onmouseover
• Added a 'skip to main content' link at top of main public pages
• Menus
  • Added new role="navigation" and aria-label to main menus
  • Added hidden h2 element to main menus with the name of the main menu
• Aggregation banner
  • Added role="banner" to the banner area
  • Also added hidden h1 element with the same information as the web page title
• Facets
  • Switched main surrounding div to nav
  • Added role="complementary" and aria*label
  • Changed alt tag on the button to sort the facets
• Child aggregation buttons on home pages
  • Added a new section tag around buttons
  • Added role and aria-label to the section
  • Removed layout table around buttons and switched to styled unordered list
  • Corrected a repeated ID tag
• Item viewer
  • Corrected the alternate text for the page viewer
  • Minor HTML optimizations around the citation viewer
  • Added role="complementary" on the left nav bar and aria*label
  • Added nav around the TOC
  • Fixed alt and title tag of wordmarks being identical
  • Removed remaining tags in citation
• Many small ALT text corrections
• Currently rendered language is now included in the top*level HTML tag

Customization

• Menus more likley to use class, rather than ID, for currently selected option
• Added new ID to the basic and full text buttons, to allow individual customization
• Basic searches
  • HTML updated to use divs, not a layout table
  • Added additional IDs in the process and ability to customize the button more
• Advanced searches
  • Added new IDs to promote increased customization, such as updating search prompt and hiding help button

Various UI fixes

• For Chrome, PDF is shown in iFrame, and source is reloaded after it is resized
• Fixed the buttons on the private and dark items screen (SOAS)
• Corrected bookshelf pop-up form styles
• Corrected behavior of the 'remove' link in the bookshelf results viewer
• Corrected the images in the aggregation share pop*up form
• Corrected issue where Page Turner could be added (due to problem above) with no actual page images (UOC-2)
• Corrected issue where quoted search term with a space was losing the quote and being split occasionally (WOLF-25)
• Corrected a couple issues with using different search links, like OR and NOT (WOLF-16)
• Reordered some of the elements in the citation/description view (WOLF-8)
• Corrected issue with & being in the metadata fields causing the links in the citation/description to be wrong (WOLF-27)
• Corrected issue with the project template being used for standard complex metadata editing (NCF)
• Corrected issue where the clearing the committee members was causing a NULL failure (NCF)
• Physical location ( boxes, etc.. ) in citation is now a DL, rather than a layout table
• Fix to user admin - was unable to remove ability to delete all items
• Added new [%INSTANCENAME%] directive to web skins writer to add name of instance
• Updated the partner (institution) list to use unordered list, rather than layout table also
• Corrected an issue with an ID being duplicated in the instance home page with multiple themes

Installer

• Updated the sample web skin to be much better
• New web skin ( with [%INSTANCENAME%] directive ) means no need to create skin HTML during installation

Miscellaneous

• Fix to ensure the user overide QC config is read and is read when the cache is reset online
• When a user registers and requests submit rights, email to admin now includes portal name and URL

Version 4.8.9

The greatest change in this version is the addition of a new builder module that automatically processes the IIS web logs each month, storing the stats in the database and optionally sending out usage emails to everyone who has submitted material to the repository. In addition, some updates to the child page administration are included as well as updates to the template used to edit default metadata sets online. Of note, the default metadata sets can now include wordmarks, a tickler, and default aggregation links. Users can also now upload XML and KML documents at the aggregation and web skin levels.

The full updates for this release follow:

• Added new builder module to automatically add usage stats from the logs and send usage emails
• System-wide settings
  • Added Builder IIS Logs Directory setting - location of the IIS web logs for this instance
  • Added Builder Send Usage Emails setting - flag indicates is the builder shoudl send usage emails
• Update to handle missing skins folders better
• Added a new builder module to ensure all images in digital resource folder are attached to resources
  • Made new module default and unenabled the AddNewImagesAndViewsModules in db
• Added new stored procedure to set additional work needed flag on item
• Online editing of default metadata sets fixes
  • Corrected link to demo 'new element' plus sign
  • Set to use new template XML file ( project.xml )
  • New template includes behavior elements of wordmarks, tickler, and aggregations
• Metadata fix
  • Creator fix, was not mapping to MARC without a full name
• Single aggregation admin
  • Child Pages Admin
    • When adding multiple child pages, parent code shows correctly immediately
    • If a child page is deleted which has children, those children move up under the deleted page's parent
    • Child pages cannot be set to be under a child child page in the menu hierarchy
    • Fixed a hover over for deleting child pages
  • XML and KML can now be uploaded in the document upload tabs
• Single web skin admin
  • XML and KML can now be uploaded in the document upload tabs
• Added a metadata browse link to the aggregation manage menu

Version 4.8.8 (B)

Of greatest note is the addition of new item-level REST API microservice endpoints within the system. More documentation on this will be forthcoming soon.

The new version can be downloaded at: http://sobekrepository.org/software/download

The full list of modifications follows:

• Item-Level SobekCM Engine REST APIs
  • Added first item-level engine endpoints based on the BriefItemInfo class.
    • One endpoint returns just the citation
    • Second endpoint also includes all the files and table of contents
  • Added a large number of brief item mapping classes to map from the SobekCM_Item to the BriefItemInfo classes.
  • Mapping is configurable via the configuration file for the mapping sets
  • Added a new random item API for selecting a random item from the instance
  • Added endpoints to serve XML ( mods, dc, and marc ) and RDF endpoints for the items
• Reorganized REST API endpoints and finished some item-level endpoints
  • REST API endpoint URLs changed to have the protocol last, for easier management
• Automatic code generation
  • Moved the Sandcastle documentation creation project out
  • Truncated property names to 50 characters for localization files - was causing a documentation creation error
  • Corrected some errant namespace definitions that became apparent during code documenation creation
  • Other small fixes discovered during code documentation
    Top-level static HTML pages (simple CMS webcontent mode)
  • Changes to the styles of the top-level static HTML pages (for the final code documentation)
  • Updates to add menu to top-level static pages
• Metdata Updates
  • Fixed issues with handling of & in some fields
  • Name as subject was not mapping to MARC correctly, if no name type specified
  • Name as subject was not displaying correctly in description without a full name specified
  • Fixed misspelling on government publication ( terriorial --> territorial )
  • Corrected issue with main entity name not showing correctly if no full name was provided
• Item Viewers
  • Download viewer now used the file name as the link, if the group was not named
  • Update allows the item HTML viewer to load HTML from an external URL. i.e., a source that starts with http:// or https:// or <%BASEURL%> or [%BASEURL%]
  • Statement of responsibility note always appears as last note in citation/description now
• Online submission and management of digital resources
  • User's IP address is included in the agreement.txt saved when a license is agreed to while adding a new item online
  • Corrected line in agreement.txt to exclude shibboleth ID if user does not have one
  • Fixed issue adding empty creators in simple creator template ( also caused an error during marc.xml generation)
  • Adding files through page image upload, when pages exist already, behaves better
  • Corrected issue loading the group behaviors template file ( due to new users/default folders)
• Quality Control
  • Corrected some references in QC javascript that caused multiple page move to fail
  • Added some error writing when exceptions caught during QC that may result in references to items being cleared
  • When saving after QC, file count, page count, and total size are now correct
  • QC progress is now saved correctly even if other progresses from the same day exists in different workflows
• Single Item Aggregation Admin
  • Corrected issue when editing an aggregation child page that didn't allow the parent for menu-type visibile items to be correctly displayed
  • Correction to allow parent and visibility of existing child pages to be correctly edited
  • Update to start editing child pages in HTML source mode if it has scripts or inputs.
• Single Web Skin Admin
  • Now there are SAVE and SAVE & EXIT buttons, to apply changes but not leave the screen
  • Added links to copy the HTML from the standard header/footers down to the item headers/footers
• Misc
  • Removed the SobekCM_Rest_API project - not used
  • The track item mySobek viewer no longer throws an exception if no scanning equipment was entered
  • Corrected two links on the user's logged in mySobek menu ( account and my library )
  • Corrected height of selected submenus (really only affects child pages menus which show selected menu items in submenuts)
  • Version link under system admin menu now points to sobekrepository, not ufdc

This Subversion (4.8.8B) release includes a number of bug fixes found in the last version and improves support for the REST API error reporting and encoding.

• Corrected issue with (last) division not saving correctly if it was a type that did not allow user to enter name
• Applied CKEditor updateElement() fix to the child pages online editing
• Added text to the possible document types that can be uploaded to aggregations and skins
• Corrected aggregation/skin uploaded document display to include all the possible types (some were missing)
• Set engine encoding to UTF8
• Engine now overrides IIS's errors, so the engine's custom errors can be displayed
• Fixed json endpoint for images used during online html editing. Was including non-image documents, causing a CKEditor error
• Added some error catching on the aggregation services endpoints