Campus Software Development

CTSC - Campus Software Development

CTSC voted on future topics of interest, and software development was by far of the greatest interest. The original voting description said:

Software development.

The role of IT groups in the Faculties for software development and innovation.

with some follow on comments/notes:

How is it supported?
How are decisions made on what to do?
Who supports it after it's developed? Etc.
Consequences of activity based budgeting.

The 2015-05-14 CTSC minutes say:

For each of the top 6 topics we need to determine:
Champion/sponsor

Presenter(s)

Frame the discussion; what do we want out of the topic?
We need deliverables and expectations.

Deliverables
Related Problems
Some Answers

Deliverables

What Is Development ?

Development spans everything from traditional development to non-trivial spreadsheets. Tools such as Filemaker, Cognos, Access, contribute to this.

This is the classic "buy vs build" problem. We identify situations where development is more effective than buying (when buying is even possible). So development does happen across campus, and it can be beneficial.

Development includes extension of existing (open source) software.

Some of the reasons to engage in development are:

missing functionality
- UW specific behaviour, reports, conventions, formats
- constituency specific behaviour
integration
- making people's data visible to what/where it needs to be
licensing/cost
Commercial software may be quite expensive as it strives to be all things to all people, providing far more than we need. There is a tradeoff between functionality and cost (the "good enough" argument).
focused, reduces unnecessary complexity
Related to the above; the complexity of a commercial solution can impose significant learning/operational costs.
short term solutions
We can't always wait for available funding and a lengthy purchasing process.
prototype (to determine client needs)
no viable alternative
staff development, of tools and expertise

Problems With Current Development Practice ?

availability of needed skills - people with the needed skills might be available only elsewhere
we aren't sharing the results to the extent possible ?
- visibility
  - lack of an inventory of possibilities, e.g. OFIS, MyCommonCV, ...
  - lack an inventory of (who has what) skills
    - "imprinting" (becoming the visible expert) => load
  - mailing lists to help for uw-specific technology problems ?
- ongoing support, sustainability
  - lack of adequate documentation ?
  - use co-ops ?
    - learning curve
    - security
- desire for local optimization may reduce the desire to share
- sharing is difficult ?
  - mechanism for sharing
  - needed generalization
  - variety of languages/tools/libraries/frameworks/methodologies/...

How Do We Improve ?

This is a fine discussion topic for CTSC.

Some Answers

A query (from Erick) about development was sent to the CTSC membership.

Given the above observations, it may may be that "almost everyone" develops. At very least, these groups have admitted to development:

AHS — 'minimal' (1→2 people)
Arts Computing — yes (3 people)
CSCF — yes (2 people)
EngComp — yes (3 people)
IST - Client Services — yes (10 people)

We have yet to hear from:

This observation will be removed when this leaves CTSC.

Arts (Dawn Keenan) suggests:

One question I would add to the list is whether (and how) your team collaborates with other units in the creation and management of locally developed software.

While some software is specific to a given org unit's needs, others can be used effectively in other areas, often with minimal modification, to address similar needs. Not all such development is well suited to the global open source community.

Who Develops and Why

Does your team do any software development?
Why or why not?
Do you wish you responded the other way?

AHS

Minimal. AHS does not have the manpower to do this? The IT staff are not programmers. I'd be happy to have development staff on the team but can't sell it to the Dean.

Arts

We develop software when it is the best approach to solve a problem or support a specific function.

CSCF

Yes, we engage in software development. We do it to faciliate CS priorities. In general, it's to improve workflow, both within CSCF, and within CS in general.

EngComp

Usually to improve service to either our faculty members, our students, or the support staff and their operations where development gives us some identifiable benefits.

We have a particular audience in mind and make a business case for why each project is important. Often the project grows beyond the initial use cases.

Projects often last for years as they gain new niches.

IST - Client Services

Yes, we have a web development unit within Client Services that develops and supports WCMS, Portal, and Open Data.

IST - Security

Yes.

IST - TIS - Infrastructure Applications

Yes. The development was required to improve the workflow of vendor applications, to integrate vendor applications with local systems, to add simple monitoring tools.
We have no regrets for undertaking this development.

IST - TIS - Networks

Yes, when we think up small applications to improve our processes we need a dedicated programmer to implement them.

Library - Digital Initiatives

Does your team do any software development?
Yes.
Why or why not?
To provide needed tools and solutions or to implement, integrate, and modify adopted tools and solutions. Deploying many Open Source and Vendor tools requires many of the same skills that developing home-grown solutions require. For instance, in deploying the Open Source DSpace software we have done modifications and extensions requiring writing Java, SQL, JavaScript and other languages.
Do you wish you responded the other way?
No. I wish we could do more, but resources (programmers) are limited.

MFCF

Yes, MFCF does software development.
We started long before there were plenty of open-source or commercial tools out there. We have some needs for which there might not be closely matched tools available elsewhere.
No, we don't wish we answered the other way.

Staffing

How many of your team regularly do software development?
Is it important to their job function?

AHS

1 → 2.

Arts

The answer depends on the definition of software development. All technical staff develop and maintain tools and reporting infrastructure to improve our ability to do our work. If we are focusing on software that is readily recognized as applications rather than systems infrastructure, software development is a critical part of job function for 2 staff in the Faculty IT unit and 1 IT staff person elsewhere in the Faculty.

CSCF

We have 2 staff for which development is an important job function. Some other development happens as well (less than an FTE's worth).

EngComp

About 3 staff do much software development (Steph, Jack, Erick). Several other staff do some development, usually to address needs that arise in their normal job function.

IST - Client Services

WCMS team: 6 developers
Portal team: 3 developers
Open Data: 1 developer

IST - Security

2/3 of my team write code.

IST - TIS - Infrastructure Applications

How many of your team regularly do software development?
One.
Is it important to their job function?
Yes.

IST - TIS - Networks

Just one, our co-op, it is the major part of the job function.

Library - Digital Initiatives

Of the 5 currently working in our Digital Initiative group, 3 primarily do development work.

Our Library Technology (LTFS) group also does many tasks that include development activities, although they more administrate our hardware, software, and network needs, including servers, workstations, presentation rooms, mobile and laptop devices, and our virtualization infrastructure.

Is it important to their job function?
Absolutely.

MFCF

This number has been dwindling over the years. Our primary maintainer and developer of in-house software has recently retired; it was central to his job function. We have only a few left who do this now. There's far more expertise with shell scripts than we used to have with traditional Unix system programming in C.

What's Developed

E.g.

web apps

system tools

end-user apps

research-related

administrative related

student related

etc.

AHS

A few web apps; simple matlab code; mostly research related.

Arts

web applications for technical and non-technical audiences
applications and system tools for internal use and use by technical staff outside the unit
software and systems in support of administrative business functions
there has been some development focused on mobile devices and with a student audience in mind in the past
IT staff outside the unit write software in direct support of research

CSCF

web apps - almost entirely.
system tools - workstation and cluster setup aids.
end-user apps - the web apps are almost entirely end-user.
research-related - we do some cluster setup programs to help us setup research clusters.
administrative related - the majority involves support of some form of administration.
student related - we have local software to manage accounts.
etc.

Teaching support is much of it. After that, administrative support for faculty, the biggest being that used to assist in hiring.

EngComp

web apps - OFIS, MyCommonCV
mobile - eg. get keycodes for grad lab door
system tools - several in the pipe, plus long history with Nexus
end-user apps - OFIS
research-related - not terribly often
administrative related - often, incl. sharepoint based automation
student related - lots, licensing other things prohibitive or restrictive
etc.

IST - Client Services

Primarily web apps -- WCMS, Portal, Open Data.

IST - Security

Mostly tools/scripts to do their jobs and/or integrate with other tools (commercial or open-source).

IST - TIS - Infrastructure Applications

Web apps (instructor-, student-, administration-related), integration (system tools), monitoring (system tools).

IST - TIS - Networks

Web-apps are what we have developed.

Library - Digital Initiatives

Web apps or enhancements to websites are the primary development activity. This includes many of the areas listed: system tools, end-user apps, research-related, administrative related, and student related. Data feeds, APIs, and Web services are becoming a more prominent part of the landscape.

MFCF

Primarily system administration tools.
Some automatically generated web content.
Some automatic data collection and processing (e.g. usage stats).

Value

What value does development bring your org unit? How is this better or worse than purchasing software?

AHS

N/A.

Arts Computing

Local development helps us understand processes and needs within the Faculty, provides opportunities for professional development, and results in documented software and systems. Purchasing software can have different costs (higher or lower for both acquisition and maintenance), may require changes in processes or terminology, and is usually a faster means of providing a tool fit for a specific use.

CSCF

Workflow of both CSCF and various aspects of CS is improved. There can be a reduction in paper. And we can facilite work that might not otherwise be practical.

Local development can provide benefits beyond that which can be purchased. Buying is best when the product very closely meets needs, and doesn't have a significant maintenance cost.

EngComp

We can specify exactly what we want, and integrate it with other things on campus. Eg. flexibility with asset management to use existing campus sources. Also, OFIS deals with campus data, Engineering processes, etc.

Services can be configured with all the defaults we want, no user configuration required, no changing business operations to meet external naming conventions or different processes unless FoE agrees.

Often saves money (no restrictions on how we use it), often lets us offer new and wonderful features we wouldn't be able to afford to outsorce. Sometimes helps reputation to not be using externals (eg. survey monkey).

We still buy many things. Like we buy sharepoint, development kits, MatLab, etc.

IST - Client Services

Within the Web Development team, we develop our services ourselves as it allows us to better meet our client's needs, support our services in a sustainable way, and invest in our people. We do occasionally contract out work to web development companies to build features on our platforms when it is more feasible to do so.

IST - Security

Ideally, we purchase or use open source. Sometimes, there's nothing out there that meets our needs so we need to write stuff. Most of what we write is to enhance something, or integrate things.

IST - TIS - Infrastructure Applications

Improved workflow for application users, better monitoring information. Purchased software does not exist in these cases.

IST - TIS - Networks

We value the work, we wouldn't have it done otherwise. In our case, we are typically unaware of an application that will fit our process exactly, having one developed to our requirements gives us what we want.

Library - Digital Initiatives

In our environment this is not exclusively one or the other but a necessary blend. The size of our team is too small to develop some of the software needed. Implementing an existing solution can save time and effort. For smaller projects, it is often very quick to implement something fairly simple locally. We know our environment better than any vendor does.

MFCF

It does what we need it to do.

Open Source

How does open source fit in with your development or lack of development?
Do you contribute back?

AHS

N/A.

Arts

Although we use open source software in our work, we do not at this time actively contribute to the code bases. Open source can be less expensive than either locally written or commercial software, though the total cost of ownership and the fitness of tools for the job need to be factored in to a full cost-benefit analysis.

By contrast, we are open to collaborating between units on software, tools, and configuration that has been developed by UW IT staff in any area including our own. The largest problem in collaborating among units is communication-based: updates are difficult to share and put into production despite the availability of repositories such as git.uwaterloo.ca. Initial interest in working together is frequently swallowed up by day-to-day work and local units' priorities.

CSCF

We generally use open source tools (where practical). We don't contribute back.

EngComp

We use a great deal of Open Source, for cost and flexibility.

We connect our sharepoint to both MS and MySQL databases and have for years. Allows us to integrate data and processes from different development projects.

IST - Client Services

WCMS is an open source project, built as a Drupal distribution. You can see and contribute to our codebase on https://git.uwaterloo.ca. We contribute code back to Drupal whenever possible, some members of our team our very involved in the Drupal community and maintain a number of contributed Drupal modules. We believe in developing in the open and contributing back.

Open Data API is also open source and is available at https://github.com/uWaterloo.

Portal is not entirely open source but uses some open source frameworks.

IST - Security

Open source is just another source of software. We contribute back, where it makes sense.

IST - TIS - Infrastructure Applications

There is no open source software for these cases.

IST - TIS - Networks

I am not aware of us using or contributing to open source development.

Library - Digital Initiatives

For library and academic needs, open source solutions are often developed sooner than a vendor can bring something to market. We use both vendor-provided and Open Source software. Quality and responsiveness to our needs vary widely from vendor to vendor and among Open Source projects.

Another complication in our environment is our partnership with the University of Guelph and Wilfrid Laurier University libraries. We often use the same software and share knowledge, experience, and even human resources.

MFCF

We prefer open source whenever we have to use a product.
We have contributed back a fair bit to SaltStack.

Technology

Which languages, frameworks and tools do you use most? Which OSs or Web do they support?

AHS

N/A.

Arts

At present Laravel, Rails, and Powershell are most commonly used with Apache on Ubuntu Linux and Windows Server 2012.

EngComp

Web: php, javascript (use HTML5 Builder, a commercial development system with open source RPCL libraries)

Windows: powershell (sharepoint, workstations)
Unix : php, C, csh

Windows/OSX/Android/iOS : Delphi

CSCF

We use:

Languages: Perl, Python, PHP, SQL
Libraries: JQuery
Frameworks: Django, Yii
Databases: MySQL, PostgreSQL
Tools: Selenium, Sphinx, SchemaSpy, rcs, git
Other: HTML, CSS
OS/Web: Ubuntu with Apache

IST - Client Services

Primarily LAMP stack, PHP/MySQL, Drupal.

Portal is built on .NET stack and uses Angular.JS and Bootstrap on the frontend.

IST - Security

Python, perl, ruby, shell scripts.

Note, some of us review code, so other languages such as PHP are known written. I write lots of SQL myself, for example.

IST - TIS - Infrastructure Applications

C# using Visual Studio, PowerShell, vbscript. These are Windows Server tools, and Windows IIS web sites.

IST - TIS - Networks

PHP, MySQL, jQuery, JavaScript to work with Windows/Mac using Chrome/Firefox.

Library - Digital Initiatives

PHP and Python are the most used languages. We do work in Drupal for our participation in the WCMS project (we contribute modules). We use GitHub and Jira. Most of our development work in on Linus with Apache. We are putting considerable effort into Java and Tomcat-based software projects recently.

MFCF

Shell scripts, perl, python, C, php; Django, Salt. Primarily Unix. Some Windows and Mac. (Don't understand the question about which "Web" do they support. Do you mean web browser?)

Collaboration

Whether (and how) your team collaborates with other units in the creation and management of locally developed software.

CSCF

We maintain the source for the Service Tracking system used by MFCF and CSCF.
We maintain the source and provide the WWW service for the inventory system used by MFCF and CSCF.
We once participated in the Engineering OFIS project.
We maintain and develop a branch of the (Management Science) OFAS project (for faculty recruiting). There is no ongoing collaboration, as they didn't want our updates.
We use the (Science) course evaluation system (for all non-paper student course evaluations). We have made some contributions to the codebase. In addition, they have added CS-specific formulas to their reports, as they have done for other groups.
CSCF created, maintains, and develops the Examination Management System. It provides examination seating and printing to a few groups outside of CS.
CS created, develops, and jointly manages the OAT system.

CTSC - Campus Software Development

Table of Contents