GridPP-2

Log Book

Application Middleware
Metadata

	Document identifier:	GridPP2/Middleware-LogBook-Metadata	
	Last Updated:	21st October 2004	
	Work Area:	Metadata	
	Partners:	Bristol, CERN, Fermilab, Grenoble, Manchester, Oxford, RAL  	
	Lead Partner:	Glasgow	
	Document status:	LIVE - ONGOING	
	Editor:	Tony Doyle	

Document Change Log	
Revision	Date	Comment	Editor	
1.0	16th July 04	Baseline version of logbook	Robin Middleton	
2.0	21st  Oct 04	Q3 2004 release version	Tony Doyle	

				

				

				

				

				

				

				

1Introduction and Instructions
This Log Book is designed to be the least onerous method of recording and tracking deliverables in GridPP2. It will be used initially to finalise the work programme and then to monitor progress on a quarterly basis. It will be maintained as a (suitably protected) document on the Web. 
The existing deliverables should be partitioned into two levels: 
-Level-1 deliverables that will be included in the GridPP2 ProjectMap, and which will be monitored on a quarterly basis. For each of your main objectives we anticipate Level-1 milestones at about 6 monthly intervals. 
-Level-2 deliverables that are at a more detailed level. It is not our intention to micromanage the individual tasks: These will not be monitored on a quarterly basis but may be used internally by the work-area managers to guide progress and anticipate problems.
The Level-1 should be kept up to date in this document and the GridPP Project Manager should be notified of changes to Level-1 deliverables (only) by the completion of a ChangeForm as in GridPP1. 
The Level-2 deliverables should be available as a link from this document but may be in whatever format is most suitable to the particular area (it may be a Link to an EGEE document, for example).
Quarterly reporting will be performed by the work-area manager.

2title and Activity Summary
This section should give a brief overview of the activity.
2.1GridPP2: Grid Data Management - Metadata
The following document describes the activities based at Glasgow University to develop metadata for the GridPP2 project. The group arose from earlier work on Grid Data Management (see ) on Grid Metadata Management (Spitfire) and Query Optimisation (Optor) as well as SAM software development and incorporates a new activity to apply this expertise to ATLAS requirements via the existing ATLAS Metadata Interface (AMI).  
The ATLAS (CDF) posts will enable the metadata requirements of the ATLAS (CDF) experiment to be met by developing services for data set handling, analysis and job handling requirements on the Grid. Key tasks are to:
a.gain a conceptual understanding of the existing ATLAS (CDF) metadata structures and the ATLAS (CDF) specific use cases that drive them;
b.develop, with reference to the use-cases and interactions with other ATLAS (CDF) developers, the metadata necessary to support the navigational use cases. Both the schema itself and the optimal location of the metadata require study;
c.understand the analysis use-cases and optimise the event to file granularity for different types of analysis data depending upon the use case. Develop automated ways to monitor the best granularity of event data based on analysis access patterns;
d.implement fully working and documented solutions, working with the ATLAS (CDF) team to ensure that the developments here are fully integrated with the rest of the ATLAS (CDF) software, in particular, with the AMI (SAM) products. 
The Generic Metadata postholder will work in the context of the EGEE data management project where the key objectives are:
a.development of Grid technologies within a service-focussed architecture (such as OGSA) for use in metadata based applications for the experiments;
b.delivery of fault-tolerant, reliable and manageable software for this purpose. The emphasis from the beginning will be upon developing services that meet the requirements of the experiments;
c.use of this technology for the enabling of existing experiments’ metadata based products in line with the Metadata Catalog service described in the ARDA document;
d.participation in the Grid Forum data areas to ensure that particle physics is in a position to benefit from developments here such as OGSA-DAI. Promising developments will influence the design of the metadata services and we will feed back our requirements and experience into these forums.
The work is conducted in the context of the wider forum of the UK MetaData group including representatives from many HEP experiments who are actively working in this area. 

3High Level Objectives and Level-1 Deliverables
For each High Level Objective (there may only be a single objective) please provide:
(a)Descriptive Name.
(b)Why (purpose)?
(c)Principal clients / stakeholders (e.g CMS, EGEE, User Board, Grid Deployment Board…, other).
(d)Definition of successful achievement of objective (including metrics where possible).
(e)High Level Risks to achievement of this objective. (Please review current Risk Register and use existing Risk Codes where appropriate).
(f)List of Level-1 deliverables associated with Objective: Description, due Date and Metric.
-These deliverables should be at something like 6 month intervals.
-Deliverables should be such that the PMB can see that you are on track to achieve objectives.
-Deliverables should define useful output (ideally a mixture of prototypes, services, and documents). 
-Deliverables should be manifestly something useful to your main client as listed above.
-Any specific risks (not covered above) and dependencies of individual Level-1 deliverables should be noted
No:	Project Month	Post	Deliverable	
Metric	
1	3	All	Review and assess all known documents referring to HEP metadata use cases	Document	
2	6	All	Documenting schema for metadata	Document	
3	8	All	Definition of Metadata service (including distinction by experiment)	Document	
4	33	All	Implementation of appropriate metadata	Contributions to prototypes and services	
5	11	All	Document common monitoring requirements	Document	
6	12	All	Assisting wider adoption of Metadata service (to be reviewed annually)	Annual Review	
7	16	All	Review Metadata service definition		
8	19	All	Throughput testing of database access via Metadata	Test prototypes and services	
9	28	All	Contribution to EGEE Development	Recognised EGEE contributions	
10	30	CDF	Produce documentation on distributed databases	Document	
11	24	All	Produce documentation on user access optimisation of data access via metadata	Document	
12	36	All	Final evaluation of metadata components	Document	

4Level-2 Deliverable or Milestone
A more detailed execution plan than the Level-1 deliverables is required. However, it may be maintained in whatever format is more suited to each area. Either a link to something that shows the current plan/status should be contained in this document or the plan should be included directly.

Changes to level-2 deliverables do not need to be reported to the GridPP project manager but the document describing them should be kept up-to-date. Progress on level-2 deliverables will be regularly monitored by the appropriate PMB member (Applications Co-ordinator; Middleware Co-ordinator; Deployment Board Chair).

The type of information needed is the following:

(a)Level-2 deliverable description and due date.
(b)Metric (what defines successful completion?)
(c)Associated Level-1 deliverable.
(d)Current Status.
(e)Risks (optional) and dependencies.
4.1Level-2 and level-3 deliverables 
The following outlines the deliverables for the GridPP Metadata posts agreed within the GridPP project.  These are focused upon: the ATLAS experiment; the CDF experiment; generic metadata.  Level-1 deliverables are displayed with a whole number, n, and their associated Level-2 and -3 deliverables are shown underneath as n.1, n.1.1 etc. 
The deliverables are shown in their intended order, with the postholder primarily responsible indicated. Italics indicate activated deliverables. Bold indicates completed deliverables. 
No:	Project Month	Post	Deliverable	
1	3	All	Review and assess all known documents referring to HEP metadata use cases	
1.1	2	ATLAS	Review and assess ATLAS documentation referring to HEP metadata use cases	
1.2	2	CDF	Review and assess of CDF documentation referring to HEP metadata use cases	
1.3	3	Generic	Review and assess documentation referring to HEP metadata use cases	
2	6	All	Documenting schema for metadata	
2.1	5	ATLAS	Document AMI schema for metadata	
2.2	5	CDF	Document SAM schema for metadata	
2.2.1	4	CDF	Document existing Dimension Editor	
2.3	6	Generic	Summary document of differences in schema for metadata between experiments	
3	8	All	Definition of Metadata service (including distinction by experiment)	
3.1	7	ATLAS	Definition of Metadata service in AMI	
3.2	7	CDF	Definition of Metadata service in SAM	
3.3	7	Generic	Architecture and Planning Report	
4	33	All	Implementation of appropriate metadata	
4.1	9	All	Initial minor release	
4.1.1	4	ATLAS	AMI Admin Tool	
4.1.2	5	All	Review of EGEE Metadata Catalog	
4.1.3	6	Generic	Design of Grid Services	
4.1.4	6	CDF	Develop Dimension Editor	
4.1.5	8	All	Develop prototype Metadata Catalog	
4.1.6	9	All	Software and associated documentation	
4.2	21	All	1st Major release	
4.2.1	20	ATLAS	User Metadata writing capability code and user documentation in commisioning run	
4.2.2	24	All	Prototype distributed metadata service with GANGA interface, delivery middleware and web services interface.	
4.2.3	18	CDF	Optomised  Dimension Editor	
4.2.4	18	Generic	Review Architecture and Planning Report	
4.2.5	20	Generic	Design and Refactoring of Grid Services	
4.2.6	21	All	Software and associated documentation	
4.3	33	All	2nd Major release	
4.3.1	33	ATLAS	Production release	
4.2.2	30	CDF	Fully functional Dimension Editor	
4.3.3	30	Generic	Review Architecture and Planning Report	
4.3.4	32	Generic	Design and Refactoring of Grid Services	
4.3.5	33	All	Software and associated documentation	
5	11	All	Document common monitoring requirements	
5.1	10	ATLAS	Document monitoring requirements of AMI	
5.2	10	CDF	Document monitoring requirements of SAM	
5.3	10	Generic	Document common monitoring requirements (using AMI and SAM documents)	
6	12	All	Assisting wider adoption of Metadata service (to be reviewed annually)	
6.1	o	All	Maintain collaborative resources ( to be reviewed quarterly)	
6.1.1	o	Generic	Wiki	
6.1.2	o	ATLAS	GridPP Metadata web pages	
6.1.3	o	Generic	CVS repository	
6.1.4	o	Generic	Defect tracking	
6.1.5	o	CDF	Other resources for collaborative work	
6.2	12	All	External Presentations	
6.2.1	12	All	Submit documentation to the GGF-PNGA working group in GridForge	
6.3	12	All	External Project Integration Report (Confidential)	
7	16	All	Review Metadata service definition	
7.1	14	ATLAS	Review AMI implementation	
7.2	14	CDF	Review SAM implementation	
7.3	15	Generic	Identify generic components	
8	19	All	Throughput testing of database access via Metadata	
8.1	18	ATLAS	AMI throughput testing of database access via Metadata using AMI	
8.1.1	18	ATLAS	Evaluation of AMI Metadata service from DC3	
8.2	18	CDF	SAM throughput testing of database access via Metadata using SAM	
8.3	18	Generic	Common testing interface	
9	28	All	Contribution to EGEE Development	
9.1	4	ATLAS	Initial EGEE Phase 1 release	
9.2	16	CDF	Final EGEE Phase 1 release	
9.3	28	Generic	Initial EGEE Phase 2 release	
10	30	CDF	Produce documentation on distributed databases	
10.1	28	ATLAS	Review POOL and RLS for DD (ATLAS)	
10.2	28	CDF	Review SAM and RLS functionality	
11	24	All	Produce documentation on user access optimisation of data access via metadata	
11.1	23	ATLAS	Produce a document on user access optimisation of data access via metadata in AMI	
11.2	23	CDF	Produce a document on user access optimisation of data access via metadata in SAM	
11.3	23	Generic	Produce generic document on user access optimisation of data access via metadata	
12	36	All	Final evaluation of metadata components	
12.1	35	ATLAS	Final evaluation of ATLAS metadata components	
12.1.1	27	ATLAS	Evaluation of ATLAS metadata services in commissioning run	
12.2	35	CDF	Final evaluation of CDF metadata components	
12.3	35	Generic	Final evaluation of metadata components (summary)	

5Commentary
This section is filled in incrementally quarter by quarter as a means of documenting particular successes, failures, issues, problems and their resolution. It should be brief, but should provided a coherent record of the evolution of the work. It will be reviewed each quarter by the chair of the relevant board and by the Project Manager. It may be a hyper-link to an external document such as an EGEE quarterly report or a collaboration report. However, it should state explicitly which level-1 deliverables have been completed in the quarter and should comment explicitly on any level-1 deliverables that are overdue. In this case, a modified date should be agreed and a Change form should be sent to the Project Manager.
5.104Q3 REPORT
5.1.1Status of ATLAS postholder deliverables 
Generic
- A survey of all available HEP metadata / Grid use cases has been carried out and a common ground established. The "Core HEP Use Cases for Metadata" document is on target for release by the end of October 2004. 
(Deliverable 1 - 1.1, 1.2, 1.3)
- GridPP Metadata web pages have been updated and are being maintained.
(Deliverable 6 - 6.1)
ATLAS
- Links with ATLAS metadata developers have been established.
	- Knowledge has been gained of the workings and structure of AMI code. (Deliverable 4 - 4.1)
	- AMI installed on a new machine. (Deliverable 4 - 4.1)
	- AMI development database and web service established in Glasgow. (Deliverable 4 - 4.1)
	- Glasgow secure CVS now used as AMI repository to aid collaboration. (Deliverable 6 - 6.1)
	- Meeting held at CERN during ATLAS Software Workshop.
- ATLAS collection-level use cases have been developed, informed by use cases from the ATLAS Distributed Analysis group and by efforts on common use cases. (Deliverable 3 - 3.1)
- Effort has begun on developing a gLite Metadata Catalogue compliant interface to the ATLAS metadata system, AMI. (Deliverable 4 - 4.1)
- Supported and advised Chris Collins-Tooth in building an interface for AMI embedded in the GANGA job submission client. (Deliverable 4 - 4.1)
Talks: (Deliverable 6 - 6.2)
14/9/04     Metadata          				GridPP11     	Liverpool
23/9/04     ADA Catalogs in AMI (with S. Albrand)    ATLAS Software    CERN

5.1.2Status of CDF postholder deliverables 
1.2 Review and assess CDF documentation referring to HEP metadata use cases.
CDF5858, and use cases from Rick St Denis are both on the Wiki.  A CDF summary document is being compiled to meet the internal milestone.
2.2 Document SAM schema for metadata.
The SAM schema for metadata is available on the Wiki. Documentation is in preparation for each of the areas.
2.2.1 Document existing Dimension Editor
There are some documents of chains and links, and the initial Dimension Editor on the Wiki.  An overview document is in preparation.
4.1.4 Develop Dimension Editor
Development is underway.
Completed sub-tasks during the quarter:
Generic
- Reviewed Core Use Cases document.
CDF
- Set up SAM development environment on a Glasgow PC, including a miniature SAM database, working dbserver and metadata server prototype code. (Have now begun the above process on a metadata specific PC).
- Worked with SAM core developers to improve the above process and updated documentation on the web.
- Worked with Randolph Herber to produce code for the lexor/parser portion of the metadata server and added this to cvs.
6.2 Posters/Papers
First author on these papers & posters:
JIM Deployment for the CDF Experiment
>
Deployment of JIM for the CDF Grid
http://www.allhands.org.uk/proceedings/papers/85.pdf
Contributed to these papers & posters:
Testing the CDF Distributed Computing Framework
>
Deployment of SAM for the CDF Experiment
>
Tools for GRID deployment of CDF offline and SAM data handling systems for Summer 2004 computing.
>
Using SAM datahandling in processing large data volumes
http://www.allhands.org.uk/proceedings/papers/116.pdf
(Also named author on most of the CDF submissions to CHEP04).
5.1.3Status of generic postholder deliverables 
3.3  Generic Architecture and Planning Report
Initial version of high-level view of generic metadata service complete.
4.1.3   Generic Design of Grid Services
Completed development testbed machine environment for members of metadata group.
Documented testbed via wiki.
6.1.1  Generic Wiki
Wiki implementation chosen (phpWiki) and installed.
See 
User accounts created for members of metadata group.
Content added and editorial work in maintaining data
6.1.3  Generic CVS repository
Establishing accounts for members of metadata group.
Hardware support issues due to failing equipment.
5.1.4Status of Grid Query Optimisation 
During the last quarter version 2.0 of OptorSim, incorporating a new timing model making the code more than ten times faster has been developed by David Cameron (DC) and Caitriana Nicholson (CN). A new GUI was developed by an MSc(IT) student, Jamie Ferguson, during this period and has been incorporated in version 2.0. The complete LCG Grid with ~70 sites has been implemented and tested in this version by CN. The code was completed during this period but release awaits the final documentation. This is considered as the end point of a major development cycle, since DC will shortly complete his thesis and CN will be based at CERN for 6 months working in the LCG Grid Deployment group led by Ian Bird, primarily with James Casey. A poster presentation on OptorSim was made by CN at CHEP04. A talk on Grid Data Management was presented by DC at AHM04. 
5.1.5Summary
Steven Hanlon (SH) [ATLAS], Morag Burgon-Lyon (MB-L) [CDF] and Paul Millar (PM) [Generic] now form the core metadata team as part of grid data management work within GridPP2. 
The metadata group formed and proposed a set of deliverables in August 04 that were accepted in GridPP in September 04. Formal incorporation of CDF metadata deliverables has been completed. 
Work has started in each area, including evaluation of the Glite prototype and the OGSA-DAI software. 
The ATLAS postholder has been primarily responsible for the use case document, whilst the generic postholder made transition to a full-time role over the summer. 
Attention is being paid to wider aspects of CDF and ATLAS grid development, as well as incorporation of wider experiment stakeholders via the monthly UK metadata meetings organised by Rick St Denis. 
Weekly (minuted) meetings have been held throughout the quarter to review progress. Associated work on query optimisation and data management throughput testing is also reviewed and is reported here as other effort. OptorSim 2.0 has been completed and will be released shortly.
Contacts have been established within EGEE, ARDA and SAM management teams. 
Formal mechanisms for reporting have been established via the LogBook.

6Meetings, Papers & Dissemination
Please document, incrementally over the quarters, meetings attended (not internal / informal meetings), papers, posters, talks, publications and any other formal documents (e.g. user guides) produced. You should also mention any other dissemination activities (e.g. contact / working with other projects and/or industry).
6.1Meetings attended
•GridPP11, CHEP04, AHM04 and ATLAS software week (Sept 04).
6.2List of Conference Papers / Talks / Posters
The list corresponds to the subset from the Glasgow-based team. For a full set of metadata-related papers, see  with additional papers on data management and query optimisation at ,   and .

Talks
Date	Title	Presented by	Meeting	Location	
3/6/04	SAM: Metadata and Middleware Components	Morag Burgon-Lyon	GridPP10	CERN	
3/6/04	Metadata Requirements for Data Management	Paul Millar 	GridPP10	CERN 	
2/9/04	Replica Management Services in the European DataGrid Project	David Cameron	AHM04	Nottingham	
14/9/04	Metadata:Project Management	Tony Doyle	GridPP11	Liverpool	
14/9/04	Metadata	Steven Hanlon	GridPP11	Liverpool	
29/9/04	Housing Metadata for the Common Physicist Using a Relational Database	Rick St.Denis 	CHEP04	Interlaken	
Posters
Date	Title	Presented by	Meeting	Location	
September 2004	JIM Deployment for the CDF Experiment	Morag Burgon-Lyon	CHEP04	Interlaken	
September 2004	OptorSim: a Simulation Tool for Scheduling and Replica Optimisation in Data Grids	Caitriana Nicholson	CHEP04	Interlaken	
Conference papers
Date	Title	Presented at	
August 2004	Deployment of JIM for the CDF Grid	AHM 2004	
August 2004	Using SAM datahandling in processing large data volumes	AHM 2004	

August 2004	Replica Management Services in the European DataGrid Project	AHM 2004	

			

			
September 2004	Housing Metadata for the Common Physicist Using a Relational Database	CHEP04	
September 2004	Testing the CDF Distributed Computing Framework	CHEP04	
September 2004	Deployment of SAM for the CDF Experiment	CHEP04	
September 2004	Tools for GRID deployment of CDF offline and SAM data handling systems for Summer 2004 computing.	CHEP04	

September 2004	OptorSim: a Simulation Tool for Scheduling and Replica Optimisation in Data Grids	CHEP04	
			
September 2004	JIM Deployment for the CDF Experiment	CHEP04	
6.3List of publications
  Analysis of Scheduling and Replica Optimisation Strategies for Data Grids Using OptorSim. D. G. Cameron, R. Carvajal-Schiaffino, A. P. Millar, C. Nicholson, K. Stockinger and F. Zini. Journal of Grid Computing (to appear).
6.4Formal Documents
The Use Case document early draft is available for general comment via the wiki 
The associated formal document is currently being prepared.
6.5Other Dissemination activities
CHEP04 help on stand (M. B-L). AHM04 help on stand (TD). Informal metadata meeting held at CHEP04 with all metadata stakeholders.

7Effort Delivered
There should be one table per person active in the project. It is an embedded Excel spreadsheet allowing totals and checking to be done each quarter. If a person is replaced, please leave the old table and start a new one for the new person (copy and paste an existing table).
See associated spreadsheet.