MSF for Agile Software Development

MSF for Agile Software Development

Granville Miller (aka Randy Miller)

 

MSF is an agile software process that reflects some of the practices used within Microsoft.

 

Agenda

  • Agile Software Development
  • Team Model
    • Team model is the most critical piece of the agile software development process. If you know it, you can answer 90% of the questions.
  • Personas/Scenarios
  • The “Agile” Pattern
  • Shadow Applications
  • Incremental Threat Modeling
  • Context-Driven Testing
  • Risk

Randy usually gives people about a day’s worth of presentation on this stuff, and we’re trying to cover it in an hour, so this will be much abbreviated. There’s much more info available on Microsoft’s Web site (see link, below).

 

Project Success is Rare

  • 2004: 15% failed, 51% challenged, 34% succeeded
  • 2000: 23% failed, 49% challenged, 28% succeeded
  • 1995: 40% failed, 33% challenged, 27% succeeded
  • 1994: 31% failed, 53% challenged, 16% succeeded
  • The good news: State is Improving
    • Small Releases
    • Increased Customer Involvement
    • You’d think success would depend on some new server technology or something, but customer involvement seems more important
  • Agile Manifesto
    • Working software over comprehensive documentation
    • Customer collaboration over contract negotiation
  • An agile method is
    • Iterative
    • Incremental
    • Self-organizing and
    • Emergent.
    • It must include the attributes; otherwise it is a “lightened defined process”.

Activities in MSF

  • There are more roles than just customer and developer. There are architects, testers, etc., who add value.
    • More roles -> communication complexity increases.
  • Composed of 14 basic work streams
    • Basic activity building blocks of MSF
    • A work stream is an activity that is composed of other activities
    • Activities are described using the ETVX format (whatever that is)
    • Contains 70 activities (not including work streams)
    • Most work streams are performed by a single role
  • Roles <-> Constituencies
    • Team of Peers
    • Business analyst: Product management
      • is responsible for two constituency groups: one is the sponsor, the other is the users.
      • is responsible for tracking time.
  • Release manager: in charge of understanding deployment, handing it off. Product manager: responsible for tracking budget and such. They can be combined.
  • Roles
    • Developer
    • Project Manager
    • Business Analyst (handles interaction with customer)
    • Architect
    • Tester
    • Release Manager
  • Work Streams / Goals
    • Capture Product Vision (business analyst)
    • Create a Scenario
    • Create a Quality of Service Requirement
    • Plan an Iteration (project manager)
    • Create Solution Architecture (architect)
    • Implement a Development Task (developer)
    • Build a Product
    • Test a Scenario
    • Test a Quality of Service Requirement
    • Fix a Bug
    • Close a Bug (tester)
    • Release a Product
    • Guide Project (project manager)
    • Guide Iteration (project manager)

Safe Harbor Statement

  • Choose the MSF for Agile Software Development process for projects with results-oriented teams who can work without lots of intermediate documentation
  • “Stretch to Fit” instead of “Shrink to Fit”
    • Minimalist approach
    • Agile methodologies promote adaptation

Scaling to All Project Sizes

  • There’s a matrix for whether it’s a good fit to combine particular roles
    • (P)ossible, (U)nlikely, (N)ot recommended
  • “If you have the skills to play a role, you may play that role”
  • Smallest project size is 3 people (with one or two people, you probably don’t need much process)

Mindsets:

  • More than a set of activities – a collection of values
  • Quality is Defined by Customer
  • Pride of Workmanship
  • Team of Peers
  • Frequent Delivery
  • Willingness to Learn
  • Get Specific Early
  • Qualities of Service
  • Citizenship
  • Different roles. Some care only about time. Some don’t care so much about time, and fight for quality. Who breaks deadlocks?

Adaptation

  • There are many events that may befall a project that are not covered by our process
  • Use best judgment and adapt based on mindsets and principles

Personas

  • Descriptions of a group of typical users
  • Persona represents a “proxy” for the user group, and provides a means to talk and reason about the group through the characteristics of one fictional individual
  • On-site customer is the best you could hope for.
  • Actors in use cases: the other extreme. A developer is a developer.
  • Persona: fictional representations of people.
    • Be respectful.
      • Best Buy did a set of personae for their customers. One was the “Devil Customer”. This was fine until it landed in the Wall Street Journal.
    • Can represent the novice and the advanced developer.
    • Scenarios associated with each persona.
    • Knowledge, skill, usage pattern
  • Can still meet actual customers (and refine our personae).
  • Different people come up with different personae, write them on sticky notes, put them on the board. When you take yours up, you decide whether it should be combined with one of the others, or should be separate.
  • Q: Where do you get pictures to use for your personae? A: Got any artists in your group? Can you buy a CD of stock photography?
  • Baseball cards of personae (including organizations)
  • When you’re trying to make a decision, at least you have a clue.

Jack

  • Role: Online shopper
  • Motivation: Get it quick
  • Usage pattern: Hates shopping but wants to get it quick. …

Incremental approach – Why scenarios over use cases?

  • Risk reduction
    • Changes
    • Delivery
    • Progress
  • Activities need to be able to handle small increments up to a single change
  • A scenario is basically a single path through a use case. Easy to break into smaller increments if needed.

The Agile Pattern

  • Scenario list (aka backlog) -> iteration plan
  • Peel off pieces, add them to a particular iteration
  • The things in the list aren’t actually scenarios – they’re reminders. The business analyst writes them up for the iteration.

Date or Feature Driven

  • Which?
  • Parkinson’s Law: If you give developers more time, they will find ways to fill that time. So you have to be aggressive on the date, right?
  • But not everything is really date-driven
  • “We’re going to deliver in March, even if it’s March 51st”

Minimum Acceptance Level

  • If you ask customers what they want, they’ll list off the Exciters & Delighters. (What do you want in your TV? Remote control, stereo, closed captioning, etc.)
  • But they usually won’t tell you about the Dissatisfiers (must-have requirements) (ability to turn the TV on without the remote, good picture quality, etc.)
  • First few iterations: you work mostly on the dissatisfiers, and a little on the exciters. But you can’t deliver until you make enough progress on the dissatisfiers.
  • So even if time ran out on iteration 3, you still couldn’t ship.
    • Of course, this is all pragmatic. Depends on market. If you really want to be first-to-market, you might ship anyway.
  • Even if you’re date-driven, you might not really be date-driven, because you might not be able to ship without X features.

Feature graphs:

  • Active (until the developer finishes it – doesn’t distinguish ones that haven’t yet been given to the developers to work on)
  • Resolved (goes to test: implemented fully)
  • Closed (tester can run it totally through)
  • Do a stacked area graph over time. Active=red, resolved=yellow, closed=green.
  • If you’ve still got too much red as of the ship date, the project manager and the business analyst have a problem, and need to start talking to each other.
  • If the yellow keeps growing over time, it could mean you don’t have enough testers, or that scheduling is poor (finishing too many items at once), or the test cases are failing (coders are submitting bad code and marking it as resolved, and the testers have to kick it back), or management is putting too much pressure on and the team stops caring about actual quality and becomes totally dysfunctional.

Context-Driven School

  • Value of any practice depends on its context
  • If you absolutely *must* ship by a certain date or the company runs out of money, you might be incented to ship something that’s not ready. Is that the right thing to do? It depends on how the market reacts.
  • There are good practices in context, but there are no best practices
  • People, working together, are the most important part of any project’s context.

Context Driven Approach

  • Testing that is acceptable on opne project may be criminal on another
  • Define shipping criteria based on what’s important to us
  • Release criteria
    • No Severity 1s and 2s in the shipping product
  • Testing thresholds
    • Code Coverage for Unit Test
    • Getting 60% code coverage is really normal
    • To get 80%, you start using things like mock objects, and it gets harder
    • To get 95%-100%, you start building scaffolding and bizarre stuff
    • What’s right? Answer: it depends entirely on the project.
  • Focusing on statement and branch coverage is not sufficient.
    • Missing code
    • Incorrect handling of boundary conditions
    • Timing problems
    • Memory leaks

Bug debt

  • If the developer has more than 30 bugs, it’s time to schedule a bugfix iteration, where you fix bugs (and fix bugs only)
  • Worth doing even though you take a hit on velocity, because you’ll take a hit on velocity anyway (because you either fix them as you go, or they slow down your coding or testing)

Conduct Exploratory Testing (One Heuristic)

  • Assume a persona
  • Time-box
  • Traverse the system looking for new goals, roadblocks, or improvements, keeping the needs of the persona in mind
  • Add any new bugs, scenarios, or quality of service requirements discovered using this process

Risk and Agile Processes

  • Agile processes drive out risk through short release cycles and customer feedback.
    • Requirements risk
    • Implementation risk
    • Quality risk

Governance

  • Objectives Reviewed
  • Progress Assessed
  • Test Thresholds Evaluated
  • Risks Identified and Mitigated
  • Deployment Ready

Methods of Adoption

  • Organization / Team
  • Grassroots – individuals find the practices useful and the process spreads by providing value
  • MSF provides friction-free usage that doesn’t get in the way

More info

  • http://www.microsoft.com/msf – download “MSF for Agile Software Development (Process Guidance Only)”; has much more info than this presentation
  • Will be released in book form – MS wants to ship it with Team Foundation 1Q next year

Agile Methods with Borland

Birds of a Feather: Agile Methods with Borland

Matt Gelbwaks – Borland

 

This was pretty much exclusively done as a Q&A session. We came up with questions up front, and then they tried to answer as many as they could.

 

Note: All the questions we ask during this session are going to become requirements and scenarios in Borland’s feature backlog. Cool!

 

Interesting idea: show a scatter graph of estimated vs. actual time for each story. Hopefully that scatter graph will get closer and closer to a 45-degree line over time.

 

If we have ideas for new information radiators we’d like to see, tell Borland, or e-mail an example to Dan Massey. That’s dmassey, on the obvious domain name.

 

What’s Borland’s experience with agile?

  • Borland was doing agile before there was agile: in 1992, they were already doing short iterations, etc.
  • Today, some teams are doing formal Scrum with StarTeam
  • Larger projects: “release train” (like Eclipse – I’m not sure exactly what that means…)
    • 6-week milestones
    • Some teams do smaller iterations within that
    • All requirements in CaliberRM
    • All bugs in StarTeam
    • Starting to use Tempo
  • Many of Borland’s larger clients have SarbOx requirements, and have audit requirements. Need to prove that the work is being done the way it’s supposed to be done (so some developers need to spend their time auditing instead of developing). To really do agile, they probably need to automate that.

How do we capture key design decisions in Together?

  • e.g., how do you capture the alternatives that were considered?
    • Consider: Instead of making a decision, hold off as long as possible, to show you’re thinking about it. Include all three options in the diagram, label them somehow. As you figure out what will drive the decision, you can keep notes. (last responsible moment)
    • Keep track of the set of options that are available. Don’t collapse them into an actual decision until the iteration where you depend on that decision.
  • There are some feature requests that might cover this…
  • Short-term: using a Description field to record those decisions; whenever a decision is made, it should have been a requirement, so roll that back into the requirements repository (so it may feed into other projects)
  • Could use version control to store the history of the discussion (StarTeam’s threaded discussions).
  • Use linkages between tools (e.g., links into CaliberRM)

How do you reconcile informality of agile with formal communication?

  • Agile does require discipline, and a way to be measured.
  • Use the discussion feature in CaliberRM to record the hallway discussion you just had (and why).
  • You can’t necessarily capture those hallway conversations. But when you’re having an actual meeting, bring up the requirements grid and change it right there – don’t make notes and update the requirements later.
  • Give visibility to the time you spend on formal communication.
  • Problem with discussion feature: no way to find out whether there are open questions still hanging. Is the thread closed?

Selling agile to management

  • What’s the problem you’re trying to solve? Measure it and start improving: timeboxes, testing, etc.

What do you do when you can’t find someone with the time to play the role of customer?

  • Borland’s actual clients don’t have a body to send to Borland, so Borland uses full-time surrogates
  • If you can’t even get a full-time surrogate, try doing a big investment of time up-front, using Maya or some such
  • Then use tools like CaliberRM to keep the discussion going in trickle format
  • Three axes to address – people, process, technology
  • People
    • Deliver early and often
  • Process
    • More time up front
  • Technology
    • Radiators
    • Web context
  • Possiblity #1: If there is no customer who thinks it’s worth their time, there’s no agility – and the value of the project is zero. If management isn’t willing to make a person available to tell you whether you’re on the right track, there’s zero chance we’ll deliver what you need, and this whole project is wasted effort.
    • Maybe the person you’re delivering to isn’t really your customer. They might be your end user, but they’re not your customer.
    • Go ahead and deliver the wrong thing (which it will be, since nobody ever told you what they wanted), but deliver it with panache. If you get them mad at you, maybe they’ll take a little more interest. (Nice idea if this actually worked…)
  • Possibility #2: Sit down with customer and Maya, to elicit requirements. Deliver something tangible and viewable, quickly, that shows understanding. (Maya is coming out sometime next quarter) Or otherwise find a way to show value immediately.

Second Keynote

I didn’t live-blog during the keynote yesterday, because it was an hour and a half, and I figured I wouldn’t have enough battery. Since today’s was only an hour, I fired up the laptop. Here’s the goods:

 

Announcements (DavidI)

  • Product ordering to get conference discount: http://shop.discount.com/ or go to Programmers’ Paradise (same offer either way). International orders, pick up paper form.
  • Turn in conference evaluation form this afternoon by 3:00 to get a Borland Conference T-shirt. Sorta looks like a muted, monochrome Hawaiian shirt.
  • Tonight’s party is in the same room lunches are in. (Looks like they’ll have real food this year, too!)
  • Tomorrow, starting at 1:15, Marco giving “Fun Side of Delphi” session.
  • Borland’s SDO strategy: quick recap
  • Why being a developer sucks… even though we’re gods at what we do
  • So how is Borland helping control the madness?
  • Learn about the infrastructure being developed in order to enable the SDO vision
  • Partner testimonial, Accenture

SDO Strategy

  • Getting Good at Software – at the company level
  • Controlling the forces WITHIN AND AROUND a development organization
  • Being able to answer these questions:
    • Are you doing software right?
    • Are you doing the right software?
    • How do you know?

Getting good at software means:

  • Better targeting opportunities. Management needs to be making decisions based on the most important thing for the business.
  • Increase time-to-value of software
  • Improving quality
  • Minimizing risk, maximizing predictability
  • Reducing cost

SDO is about answering:

  • Are you doing software right, *as an organization*? Right amount of QA? Capturing requirements?
  • Are you doing the right software? Nothing pisses a developer off as much as when they get jerked around by management.
  • How do you know? Measure.

Controlling the forces…

  • …that impact a developer
    • Which projects get funded?
    • How much rigor goes into that decision-making process?
    • Which engineers get deployed? Skills, experience, availability
    • The physics of scope, schedule, resources
    • The impact of change…
    • Timeliness, fidelity, and accuracy of requirements (e.g., up-front requirements documents)
    • …and the list goes on.

Top reasons why it rocks, and why it sucks, to be a developer

  • Why it ROCKS:
    • 3. Never-ending constant supply of new technologies to learn and use – enjoyable
    • 2. You can realize your ideas and dreams without the burden of physical materials – empowering
    • 1. With every single non-trivial endeavor in the world today, software is on the critical path. You get to solve tough problems for the planet, and sometimes even change the world – seriously satisfying
  • Why it SUCKS:
    • 10. Research – never enough time for the “R” in R&D
    • 9. Technology – it changes faster than I can code for it. (You get to play with those betas, but they crash.)
    • 8. Process – often makes life harder, doesn’t guide me on what’s next
    • 7. Communication – difficult to bridge cultures, language, timezones
    • 6. Bugs – not hard to fix… hard to find and hard to track
    • 5. Outsourcing – thread of job getting “shipped overseas” (or to eastern Europe, if you’re Marco)
    • 4. Schedules – never enough time, impossible to estimate time/scope/risk
    • 3. Requirements – hard to track, never enough detaiol, becomes stale
    • 2. IT Managers: if they had brains they’d be dangerous – care about the bottom line, we care about the software
    • 1. Resources – never enough… and tradeoffs? what’s a tradeoff? How often do you have a conversation with management where they make a tradeoff?
    • All of this is out of your control.

SDO is helping to control external forces

  • E.g. Management decision making
    • Management tools tied to development reality
    • Rigor in decision making
    • Good information = better decision making
  • E.g. The Garbage-In/Garbage-Out problem
    • Good requirement fidelity = higher probability of getting it right
    • High-touch, VISUAL dialog with users: a good thing.

Demos

  • Tempo demo – wizard that asks the manager questions to nail down the requirements and importance
  • Requirements elicitation – draw flowcharts, create storyboards with photos of existing customers
    • Generates requirements, test cases, etc.
    • Shows current project status automatically, so you don’t need to spend time in status meetings or even manually updating status
  • “Requirement Elicitation”
    • Where the business meets engineering
    • Get requirements out of low-fidelity text, and into visual media
    • Higher fidelity = lower rework
    • Lower rework = happier developers
  • “Portfolio Management”
    • Buzzword alignment: “IT Governance”, “Project Portfolio Management”
    • Real-time, data-driven status
    • IT decision making with DATA
    • IT decision making with PROCESS and RIGOR
    • Better decision making = less churn
    • Less churn = happier developers
  • Getting external forces under control makes more productive, happier developers
  • Bridge the language gap between developers and management

Futures: Raaj Shinde, VP SDO Platform

Shape of things to come:

  • “Common metamodel”: Software process
    • Authoring
    • Orchestration
    • Monitoring
    • Guidance
  • Hooks for other vendors’ software, e.g. Rational, Subversion

Donald Gravlin, CTO, Accenture Healthcare Practice

  • Leading health care organization’s business needs
    • Spiraling costs are forcing healthcare to be more efficient
    • Consumer driven healthcare is driving real-time information demand from the “Quicken” generation (both consumers & physicians)
    • Regional health information networks require application interop &* collaboration – a $200B software opportunity
    • HCOs require IT resources to focus on new products & services
  • Pain points
    • Rework of legacy environment required to transition toa  SOA platform
    • Lack of governance of distriuted IT development and integration
    • Rapidly changing regulatory requirements & time/cost of getting it wrong
    • Poor visibility and predictability of software will cost lives
  • What Accenture likes about Borland’s direction
    • Borland gets what it takes to be world class developers
    • Acquisition of Tempo
    • Rapid development of elicitation technology
    • Investment in building a platform

3 things you need to remember

  • SDO is relevant to developers
  • Borland is committed to delivering SDO
  • We value you, our loyal customers

Windows Vista Overview for Developers

Windows Vista Overview for Developers

Ernie Booth, Technical Evangelist

 

Not much code, but plenty of demos

 

Mostly base OS features, not WinFX

 

  • Experience
  • Platform
  • Opportunity
  • Clear
  • Confident
  • Connected (including Peer-to-Peer)

Demo: Experiences

  • Text box at bottom of Start menu – “word wheel”. Incremental search through the entire Start menu hierarchy.
  • Start menu no longer pops out – more of an auto-expand treeview, there in place.
  • “Glass”: semitransparent title bar.
    • Desktop Window Manager takes advantage of your graphics hardware, so it’s not all tying up the CPU.
    • Used to be that, when an app locked up, it wouldn’t paint. Not true anymore: the OS hangs onto an image of what your window looks like, so you no longer need to respond to a paint message to appear on the screen. Compositing is handled in the OS/hardware.
    • You can’t overwrite other apps’ part of the screen / video memory.
  • “Breadcrumb bar” at the top of the window (Administrator > Documents > Docs > Europe; click on one of the breadcrumbs to pop down a menu of its other children)
  • Treeviews now use Mac-style arrows instead of “+” and “-“
  • “Virtual Folders”: save a search query. Richer search than in the past. Rich property system – you can flag files with keywords. Devs can add as many properties as we want.
    • Q: How fast is the search, and does it use background indexing? A: Fast, and yes.
    • Outlook uses the same search.
    • Drop down the “Author” column header. Lists all the unique values. Check and uncheck them to filter. (If none checked, means “all”; otherwise just shows the ones you checked)
    • Can show a live preview in the Explorer window.
    • Q: When you do the search, are virtual folders being created automatically? A: No. You have to explicitly go to the “Organize” menu and Save a virtual folder. (Goes into “Virtual Folders” folder by default, but you can change that.)
  • Can zoom in on icons in Explorer. If you zoom far enough in on a folder, you can see the individual “sheets of paper” inside that folder, each with a live preview.
  • All files have a “Previous Versions” tab. Can restore previous versions, to the same location or a different location.
    • Q: How long do you keep previous versions? How much disk space does that take? Does it store diffs, or the entire file? Can you purge older versions? A: They’ll get back to us on that. Watch for a post on Ernie’s blog about it.
  • Transactional file system and registry. When you copy files, you can cancel it, and it will roll back. (Sweeeet.)
    • Q: It can be rolled back, but is it true transactions, i.e., will it survive a sudden power loss? A: Nobody here who knows that answer for sure.
  • File Save dialog now looks a lot like an Explorer window.
    • Treeview on the side and everything.
    • Also lets user enter property values when they save.
    • Application developer can specify which properties are visible in the Save As dialog.
    • Developer can also customize “Common Places” on the left-hand side.

Clear: Visualize, Search, Organize

  • Enable a simpler data management experience in your application:
  • Make your app’s files part of the experience
  • Most (all?) of the Explorer controls are hostable in your app
  • Search runs on ADO queries, so apps can use it
  • Windows Presentation Foundation (WPF)
    • Superior user experiences
    • Modern client platform
    • Windows SideBar
    • More productive, less intrusive
    • DirectX 10

Connected

  • Windows Communication Foundation (WCF), previously known as Indigo: Unified API for service-priented distributed apps
  • Windows Workflow Foundation (WF): Framework for building workflow into applications (whatever that means)
  • Peer to peer infrastructure for collaborative applications
    • PNRP: Peer Name Resolution Protocol – basically DNS for P2P. Can publish a “friendly name”, which is insecure and could be duplicated, or a “secure name” with a public/private key, which you can put in your vCard or whatever. No server required.
    • Q: Will P2P support Remote Desktop? A: Don’t know. Haven’t heard anything about that.
  • RSS
    • Can subscribe to feeds through your browser

Connected: Build for Mobility

  • Be a good mobile PC citizen
    • Handle transient network connectivity gracefully with the Network Awareness libraries
    • Support instant suspend and resume
    • Improve battery life with power management libraries and notifications
  • Target your app for corridor warriors
    • Enable a great TabletPC experience with the Pen and Ink API for shape recognition and gestures
    • Add robust synchronization and conflict resolution through Device Synch and Conflict Resolution
    • Build SideShow extensions to access important information without opening the laptop
      • (SideShow is a separate device you attach to the outside of the laptop)

Windows Workflow Foundation (WF)

  • Bridging human & system workflow
  • Human workflow: Participants (people, roles), flow style (flexible, dynamic), data (unstructured, documents)
  • System workflow: Participants (apps, services), flow style (prescriptive, protocols), data (structured, transactional)
  • Example: firing someone. There’s a lot of stuff you need to do, like having Legal review it, setting up the exit interview, disabling their access to the network…
  • Available now – separate download
  • Okay, but I still have no idea what this WF thing is!

Rubik’s Cube demo

  • GUI client that shows a Rubik’s cube and can scramble it
  • Service that knows how to solves it, and sends messages back saying how to solve it
  • Simulate packet droppage. It doesn’t end up solved.
  • Turn on reliability and ordering.
  • Q: Aren’t the ideas of “reliable” and “in order” redundant when you use TCP? A: Apparently they open a separate TCP connection for each separate SOAP request, so TCP reliability doesn’t even come into the picture. (Seems kinda silly.)

Confident (= security)

  • LUA aka UPA (Least User-privileged Accounts, or another acronym)
  • When Vista launches an application, it’s launched under medium privilege. (IE runs under lowest privilege by default.)
  • Can explicitly run an app under elevated privilege.
  • Built-in administrator does not use UPA at all. It’s assumed that the administrator knows what they’re doing. When you install, you’re forced to create another user, and the default Administrator is hidden. Even if you add another user to the Administrators group, UPA is turned on.
  • View the system clock. (Which you can do now, even if you’re a non-admin!) You have to click “Unlock” to switch into edit mode. A highly magic window pops up saying “<app> needs your permission to complete this action. (Yes/No)”. You can’t send messages to this window, so it should be hard to spoof.
  • If you try to do something that you don’t have login permissions for, you’re prompted for login credentials.
  • Can mark an app to run as Administrator only.
  • Q: What if you want to install for All Users? A: The OS does a lot of sniffing to detect that you’re starting a Setup app, and requests privilege elevation right there.
  • Q: Does the confirmation prompt happen automatically, or does a program have to explicitly call some new APIs? A: Not answered.

Running as non-admin

  • Some apps don’t work as non-admin because you can’t write to Programs, can’t write to HKEY_LOCAL_MACHINE, etc.
  • Vista adds “virtualization”. Can lie to the user, and lie to the application, to make it work. If the app writes to Program Files, it gets redirected somewhere else. Same with the Registry.

Object/Relational Mapping with InstantObjects

Okay, I’m finally getting a chance to catch up on my blogging. Here are my notes from the “Object/Relational Mapping with InstantObjects” Birds-of-a-Feather session this morning, led by Marco Cantù.

 

  • Object Persistence Framework
    • Software layer that persists the objects in the model into a storage medium (decouple model from storage)
    • Standard implementation of O/R mapping techniques
  • Advantages
    • Not locked into a specific vendor’s SQL dialect (database vendors are your enemies)
    • Ability to leverage the OO methodology
    • Reduction of redundancies in the application code
  • Disadvantages
    • Steeper learning curve
    • Inability to use some features of the DB server
      • Server-side programming, locking, transactions…
    • Possibly slower and memory-hungry applications

OPFs in Delphi

  • Bold (Win32) / ECO (.NET)
    • More than a simple OPF, Enterprise class, MDA and OCL support, ECO only available in .NET
  • InstantObjects
    • Open source; soft learning curve
    • Based on Scott Ambler’s approach (see www.ambysoft.com)
  • TechInsite OPF (tiOPF)
    • Open source; steeper learning curve; high rate of project activity
  • Roll-Out-Your-Own
    • Maybe not the best place to start – try something else first
    • See oodesign newsgroup
    • Joanna Carter’s Web site
      • She used to host a similar BOF at past conferences…

InstantObjects

  • Main advantages of InstantObjects
    • Nice integration with Delphi‘s IDE
    • Very flexible at the Business Object Model level
    • Supports standard data-aware controls
      • through TInstantSelector and TInstantExposer
    • Solid base of users and developers
  • …and current drawbacks:
    • Constraints on the database structure
      • Structure and names of keys and relationship tables are currently fixed
      • V2 improves things through the “external storage” feature
      • Was originally designed for Paradox tables – some stuff was stored in BLOB fields. In v2 it’s a memo with XML.
    • Data access is not always very fast
      • Again, things get better in v2 thanks to new brokers and new optimizations

Model Explorer pane in IDE

  • Add new class – looks like it may need to descend from TInstantObject
  • Can put class in new unit or in existing unit
  • Add attributes via GUI dialogs. You need to have an ID in every table, but you can add other attributes.
  • Name of database field doesn’t have to be the same as the property name.
  • Generates code with magic comments.
  • For every field, you get an attribute object (TInstantString, TInstantInteger, etc., and you can add your own). It doesn’t try to rely on RTTI, partly because you need more info than RTTI can provide, partly because you may want to create rich types (like Address that contains street address, city, etc.)
  • Creates a class with properties (with getters and setters that read/write the embedded attribute object).
  • Some extra metadata (max length of string fields, etc.) is put into a comment inside the class declaration. Not actually used directly at design time, except that it’s used to generate the persistence code.
  • The new class doesn’t have a constructor – just published fields for the attribute objects. Presumably the base class constructor uses RTTI to instantiate those attribute objects.
  • It isn’t smart enough to automatically quote field names when they’re reserved words. You’ll have to manually specify the storage name for that attribute.
  • TInstantSelector is a TDataSet descendant. Lets you pick which object type it contains. You pick the object class, not the table name.
  • Got a Stack Overflow in the demo (oops). Well, it’s a beta version of InstantObjects running on a beta version of Delphi

Developing InstantObjects

  • Project organization
    • Ethea is a company backing it
    • Marco is hosting the support site
  • Roadmap
    • Finally getting v2.0 out (still beta 1 stage)
      • Arrays of objects
    • Restructuring and improving documentation
      • Both user and developer documentation
    • Recruiting new developers and contributors
    • Getting work done on new features

Q&A

  • Q: How do you handle inheritance? Base class and descendant classes?
    A: Single table can host multiple datatypes – class name is stored in each row.
  • If you’re coming from a relational background, read Fowler’s “Analysis Patterns”. (Read it several times – it’ll be pretty scary the first time or two.)

www.instantobjects.org

Driver fiends

Sometimes I’m very, very frightened for the future of humanity.

Usually it’s when someone posts another comment to my rant about HP printer drivers. Because invariably, these commenters have a remarkable combination of skills: they cannot read, but they’re perfectly capable of typing. (Touch-typists, perhaps?)

These people come to ask me for printer drivers.

I don’t know why they ask me for printer drivers. I never claimed to have drivers. I certainly haven’t found any combination of keywords about printer drivers that will make Google return my Web site before HP‘s. (Yes, wonder of wonders, the people who built your printer do in fact also provide drivers for it. They suck, but they’re not that hard to find.)

And yet these people continue to ask me for drivers. Is someone playing some elaborate prank? Am I being punished by Phil, the Prince of Insufficient Light, for some past misdeed?

Or are there really that many idiots out there?

“We’ve all heard that a million monkeys banging on a million typewriters will eventually reproduce the entire works of Shakespeare. Now, thanks to the Internet, we know this is not true.” — Robert Wilensky

Using Together in Delphi

Using Together in Delphi (John Kaster)

 

Together for Delphi (both Win32 and .NET) and C#

  • New LiveSource class diagramming (two-way modeling)
  • New UML diagram suite – you could use this just as a diagramming tool
    • Class, Use Case, Sequence, Collaboration, State Charts, Deployment, Activity, and Component Diagrams
  • Design Patterns
    • Get started quickly with industry recognized GoF (Gang of Four) design patterns, or add your own (e.g., shopping cart)
  • Refactorings
  • Audits and Metrics – static code analysis
  • Document generation (generate documentation)

Together – Diagramming Types

  • Class Diagram (used by ECO)
  • Collaboration Diagram (shows how pieces talk to each other)
  • Use Case Diagram (avoids talking about terminology that will confuse non-programmers)
  • Activity Diagram
  • Component Diagram
  • Deployment Diagram
  • State-chart Diagram
  • Sequence Diagram

Design Patterns

  • Powerful Reuse Capabilities
  • Defined in Pattern Registry
  • Managed via Pattern Organizer
  • Supports C#, Delphi, Design Projects
  • Create/Edit Patterns
  • XML File Defines Model and Link Information

Audits

  • Source code conformity rules
  • Violations displayed
  • From Design Issues to Naming Conventions
  • Descriptions & How to Fix

Metrics

  • Evaluate Object Complexity
  • Quantify your code
  • Highlight Redesign Needs
  • Change Impact Analysis

Documentation Generation

  • UML Documentation Wizard
  • HTML Generation

Refactorings

  • Safe Delete
  • Changing Parameter
  • Creating Inline Variables
  • Extracting
    • Interface
    • Method
    • Superclass
  • Introducing
    • Fields
    • Variables
  • Moving Members
  • Pull Members Up
  • Push Members Down

If you use the Rename refactoring to rename a VCL control, it will change the Name property inside the DFM, and will also rename Button1Click methods.

 

Audits & Metrics: can save profiles for which rules you have enabled and disabled; can specify per rule whether it’s considered a warning or an error. Actually goes after your code, not the UML.

 

Q: Can we run audits from the command line, e.g., as part of an automated build? A: Don’t know offhand (the Together team is in St. Petersburg, so he doesn’t talk to them quite as often as the Delphi team). I’ll e-mail him and ask.

 

Q: Is there a way to disable an audit for a particular block of code (e.g., the way you can disable a particular compiler hint for a block of code)? A: Don’t know offhand; have to e-mail him.

 

Together diagramming:

  • “Generalize” = drag from one class to another; makes the source descend from the target
  • If you change the code, the diagram doesn’t refresh right away – looks like LiveSource isn’t really live

Documentation generation: same thing as JavaDoc or NDoc, using /// comments in your code. But includes a UML diagram with an imagemap, so you can click on a method name in the UML diagram to jump to its documentation.

 

Ten-second feature-matrix summary (not published yet, but this is supposed to be the final version and it’ll be up “any day now”): Language-neutral UML is only in Architect. Most diagrams are only in Enterprise, as are doc generation and audits&metrics. All Professional can do is create and print class diagrams.

 

I’m still not particularly clear on when, or how, I would use Together in a Delphi app…

What’s New in ECO III

What’s New in ECO III (Malcolm Groves)

 

A Brief History of ECO (since they cleverly scheduled the What’s New session for today, and the Introduction session for tomorrow)

  • Borland bought BoldSoft, ported Bold to .NET, and renamed it ECO.
  • When you move up from C++ to Delphi, there’s a lot of code you don’t have to worry about anymore – you don’t have to write message handlers anymore, etc. ECO does the same thing: raise the level of abstraction, free you up from writing plumbing code.
  • Modeling tool at design time (the Together part). ECO is a runtime framework that executes that model at runtime. Because you have that modeling info at runtime, it can provide a bunch of services for you.
  • The actual introduction session will be tomorrow.
  • ECO is a model powered framework. .NET only. First showed up in C#Builder. ECO 1.1 in Delphi 8. Delphi 2005 added ECO II with lots of new stuff: people said “I can use this for real”.
  • ECO III is available in all editions: Professional, Enterprise, and Architect. (Used to just be in Architect) Feedback from people saying this makes them dramatically more productive. Not all features are in all editions (will have list at the end of the session).

What’s New?

  • Option to create one file/unit per class (used to be one file per UML package)
    • This option is turned off by default
    • Can define the naming convention for the file
    • Potentially much nicer for team development
    • Applies to both Delphi and C#
  • ECO Package in a Package wizard (i.e., put the ECO classes in a separate assembly)
    • No ECO space, no UI
    • Create a new ECO application, add a reference to your package, edit the ECO space, add the package
    • Can have a Customer object in the package, and use it or descend from it in the EXE
  • Improved OCL Editor
    • Harder to confuse the parser
    • Editor available in Modeling surface
    • Categorized right-hand stuff
  • Domain Pattern support
    • Right-click on surface, Create by Pattern
    • Create Custom Patterns – select a set of classes, right-click, Save As Pattern; give it a name, description, etc.
  • Design surface is cleaner – doesn’t display names where there are no names, etc.
    • Auto-names associations – adds plurals when you make it one-to-many
  • Type names configuration
    • Allow user specification of type, models language independent
    • Example: if you declare an attribute as type “Integer”, it won’t compile in C#
    • Fun: if you often mistype “Integer” as “Intgeer”, you can tell ECO that’s an alternate name for the same type, and it’ll change it back to the canonical name
      • Also fun: You can do the same thing in the Code Editor with LiveTemplates
  • Overloaded OCL Operations
  • ECO Action Language
    • Derivative of OCL to allow side-effects (adds “:=”; can call methods)
    • Implement more business logic in your model, instead of in Delphi or C# code
    • Suggest using OCL when the logic is easy, and use Delphi/C# code if it’s complicated or you might want a debugger
    • Can call other methods (including methods implemented in Delphi/C#)
  • Extenders Extended
    • Supports menu items (TMainMenu, etc.)
    • EnableOCL and VisibleOCL – evaluated, must return Boolean. Think TAction.OnUpdate, but not centralized like actions – defined on each button/etc.
    • Execute Action Language
    • Invoke Debugger
  • Multiple DB support
    • Persist some classes via one PersistenceMapper and others via another PersistenceMapper
    • Combine this with Package In Package, and you can start building domain components
  • Enhanced DB reverse engineering
  • State Machine execution
    • State Diagram with Effects, Triggers, Guards, etc.
    • Define even more of your business logic graphically, and the framework executes it.
    • Define states (e.g., Requested, Canceled, Approved, Denied, Completed), then define triggers on each transition (Approve, Deny, Cancel, Complete). It knows which triggers are available based on the current state.
    • Effects are code that’s fired on a state change.
    • Guards say, “Don’t take this transition unless this expression evaluates to True.” Detects which triggers will violate guards, and will automatically disable those triggers.
    • States can also have entry actions and exit actions.
  • ASP.NET AutoForms
    • Go from model to execution in one step in ASP.NET (functionality already exists for GUI; this adds it for Web)
    • Add New > Other > ASP.NET AutoForm
    • Helps you start testing the model, see what it means. Shortens the feedback loop.
  • ECO in every version of Delphi
    • Professional: Class modeling, WinForms only (not ASP.NET), O/R mapping, OCL, XML persistence
    • Enterprise: WinForms, WebServices, WebForms, Database persistence (1 database, 1 ECO server)
    • Architect: State diagramming/execution, multi database, multi-ECO servers

Too bad. That state stuff was looking fairly interesting, but I think buying a dozen copies of Architect is probably a bit out of our reach right now. And the Professional version can’t even do databases. Sigh.

What’s New in the Delphi Compiler?

Here’s Danny‘s session, “What’s New in the Delphi Compiler”. At this point, I am actually caught up on my conference blogging. (Caught up on the first day. Will wonders never cease?)

What’s New in the Delphi Compiler

  • Major Win32 compiler work
    • Records with Methods
    • Operator Overloading
    • Class vars (per-type global storage)
    • Unit initialization consolidation
      • Large app, lots of units, lots of initialization -> loaded lots of pages from disk to run all those initialization sections.
      • When the linker does smart linking and assigning addresses, now it puts all those initializations together in one clump of code pages. Result: faster startup (on the order of 2-5%), and significant reduction in working set.
      • Next step is to put them in a separate segment, so they can be purged from the working set after startup. Not done yet.
      • Finalizations are also clumped together.
    • Inline-able magic functions (string Length(), etc.)
  • Minor .NET compiler work
    • Minor bug fixes, metadata improvements
  • .NET Compact Frameworks Enablement
    • Soften error checking for missing CLR stuff
    • Beef up class helpers

Records with Methods

type
  TBlock = record
    procedure Foo(I: Integer);
    function Bar: TBlock;
  private
    Data: Integer;
  public
    property Color: Integer …
  end;
  • In old-style objects, virtual methods were optional, so the VMT could be at a different offset in different types (if it was there at all). Difficult to work with.
  • In the Delphi language, “class” is always a reference type, for simplicity.
  • Value types: If they’re small, allocating them is fast.
  • Yes, you can reference the type within itself (see the declaration of Bar, above).
  • Private, protected, public. Probably has strict private and strict protected as well.
  • No inheritance. (So protected really doesn’t mean anything. Maybe it isn’t there after all.)
  • Default scope is public.
  • Can implement interfaces in .NET, but not in Win32
  • No published. Records have no VMT so they can’t have RTTI.
  • Can still have variant sections in the record, but they have to be at the end. May not be able to put visibility specifiers after you’ve started a variant section. Cannot put methods inside a variant section.
  • Assigning the record will copy everything regardless of visibility.
  • Note: if you put compiler-managed types in a record (string, interface, dynamic array, Variant), the compiler will generate try..finally code all over the place for you. (This is the same rules as before.)

Operator Overloading (Win32)

  • Requires value semantics. “A + B” must return a new instance. (Otherwise, who’s going to clean them up?)
  • Supported on record types only (see previous point)
  • Same syntax & rules as Delphi .NET
    • class operator Add(A, B: TQuark): TQuark;
    • class operator Add(A: TQuark; B: Byte): TQuark;
    • class operator Implicit(A: Integer): TQuark;
    • class operator Explicit(A: TQuark): Integer;
    • class operator Explicit(A: TQuark): string;
  • Implicitly overloaded, so you don’t have to use the “overload;” directive.
  • Can’t define new operators, but you can override all of the standard ones (except assignment and pointer dereferencing).
  • If you want to support all the different integer types, you have to do it yourself.
  • Commutativity is not assumed. So if you want both TQuark + Byte and Byte + TQuark, you have to define it. (Not a big deal – have one call the other, and get it inlined.)
  • Operator Explicit is the only place you’re allowed to overload on result type.
  • If TFoo defines Add(TFoo, TBar), and TBar defines Add(TFoo, TBar), the compiler will pick one. Don’t do this.
  • Implicit conversion happens when:
    • A := B;
    • Passing parameters
    • Don’t define an implicit conversion if you will lose information.
    • Will not do long-trip conversions. If A can be implicitly converted to B, and B to C, and you try to go straight from A to C, Delphi will say “Bad programmer!”
  • Explicit is for explicit casts, like TFoo(5), or Integer(MyFoo).
    • These are allowed to lose data, e.g., going from a LongInt to a Byte.
  • If two types both define the same implicit or explicit conversion (like TFoo to TBar), you will get a compiler error (ambiguous overload, or some such) at the point where you try to do the cast.
  • You cannot call TQuark.Add directly. It doesn’t exist.
  • Operators are implicitly static.
  • WriteLn(Q) uses Explicit(Quark): string
  • If you do a hard typecast on something that supports that as an implicit typecast, it’ll work. You don’t have to declare the same thing as both implicit and explicit.
  • Cast operators: Implicit, Explicit
  • Unary: Positive, Negative, Inc, Dec, LogicalNot, BitwiseNot, Trunc, Round
  • Comparison: Equal, NotEqual, GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual
  • Binary: Add, Subtract, Multiply, Divide, IntDivide, Modulus, ShiftLeft, ShiftRight, LogicalAnd, LogicalOr, LogicalXor, BitwiseAnd, BitwiseOr, BitwiseXor
  • If you define both LogicalOr and BinaryOr, the compiler will pick one. Don’t do that.
  • You can have an operator called Divide and a method called Divide. They won’t collide.
  • Operators should always create and return a new value, not modify the inputs. (Yes, this applies to operator Inc and Dec as well.)

Operator Selection

  • Expression: A + B
  • Types: TA, TB
  • Add operator is selected from the Add operators on TA and TB only, not from other classes. (Again, it’s not automatically transitive.)
  • If TA is convertible from TB, but not the other way around, TA is considered narrower, or more specific. This affects overloaded method calls.

Class vars

  • Global storage for a field within your type
  • Descendants share ancestor’s class vars
    • Not separate storage for each derived type
TMyClass = class
private
  class var SharedStuff: Integer;
public
  ...
end;
  • Cannot initialize these in Win32, yet. (No class constructors in Win32.) But you could initialize it from a unit initialization section.
  • Accessible from class functions
  • Same default initialization rules as globals: initialized to zero.
  • “class var” starts a new section. So do “var” and “type”. (“Var” and “class var” are there because of nested types in Delphi 2005 – “type” introduces a section, so you need “var” to get back out of that mode.)
  • Can have class constants as well (but not typed class constants).

Roadmap

  • These are objectives, not guaranteed feature lists
  • “Highlander” (calendar 2006)
    • Full .NET 2.0 support & integration
    • Delphi generics, class fragments, and more
    • Full .NET Compact Framework support, including VCL.NET for CF
    • Support .NET 2.0 64-bit app development
  • “Delphi Longhorn” (not the actual codename) (sometime after Highlander)
    • Win32 & .NET development for Windows Vista OS
    • Avalon (WPF)
  • Delphi Win64
    • Closest logical evolution of Win32
    • Same shortcomings as Win32 (DLLs, APIs, security)
    • Unicode VCL
    • Company green light, just needs time & resources