RSS Feed

Understanding Jest Mocks

Sun, 25 Jul 2021 00:00:00 GMT

Jest is a powerful toolkit for testing JavaScript and TypeScript code. It's an all-in-one test runner, mocking framework, and assertion library. Without Jest, you might need to stitch together a separate library for each of these testing needs.

But I've always struggled to navigate Jest's complex mocking functionality. A long list of factors determine whether Jest replaces a module with fake logic (mocked) or executes the actual module (unmocked). To figure it out, I created a test project.

For the following questions, imagine you want to test a file which imports some-dependency — which could be a local file, an npm package, or a built-in Node.js module.

1. Explicit unmock

Does the test call jest.unmock('some-dependency') or jest.dontMock('some-dependency')?

If no, continue to the next section.

If yes, some-dependency will not be mocked.

2. Explicit mock

Does the test call jest.mock('some-dependency') or jest.doMock('some-dependency')?

If no, continue to the next section.

If the test uses the mock functions API like mockImplementation(), mockReturnValue(), or mockResolvedValue() to specify the mock's behavior, some-dependency will use that behavior.

Or if the jest.mock('some-dependency') / jest.doMock('some-dependency') call includes the factory parameter, some-dependency will use the behavior specified by factory.

Or if some-dependency has a manual mock (a file in the appropriate /__mocks__/ directory), some-dependency will use the behavior specified by that manual mock.

Otherwise Jest will autogenerate a default mock of some-dependency using the substitution approach described in the docs for jest.createMockFromModule().

3. Core module

Is some-dependency a core Node.js module?

If no, continue to the next section.

If yes, some-dependency will not be mocked.

4. Automocking

Is Jest automocking enabled? Automocking may be enabled with automock: true in the Jest configuration, or by calling jest.enableAutomock() in the test.

If no, continue to the next section.

If some-dependency has a manual mock (a file in the appropriate /__mocks__/ directory), some-dependency will use the behavior specified by that manual mock.

Otherwise Jest will autogenerate a default mock of some-dependency using the substitution approach described in the docs for jest.createMockFromModule().

5. Local module

Is some-dependency a local file?

If no, continue to the next question.

If yes, some-dependency will not be mocked.

6. Manual mock

Does some-dependency have a manual mock (a file in the appropriate /__mocks__/ directory)?

If no, some-dependency will not be mocked.

If yes, some-dependency will use the behavior specified in the manual mock.

Faster Google Classroom Assignments with a Bookmarklet

Mon, 01 Feb 2021 00:00:00 GMT

Photo by Christin Hume on Unsplash.

If you're a teacher and manage a Google Classroom, you probably have daily or weekly routines for adding assignments. If you frequently assign work to a subset of your students, like "Advanced Readers" or "Group Two", here's a free tool that can streamline your work and avoid repetitive tasks.

I wrote this tool for my wife to use during Distance Learning, and hope it will benefit other teachers too.

Browser Bookmarklets

Most browsers support "bookmarklets" as a lightweight way to script interactions with a webpage. You're probably familiar with "bookmarks" as a means to save a webpage address so you can find it later. Bookmarklets live alongside your bookmarks, but instead of navigating you to a saved address, a bookmarklet can perform actions on the current page.

A bookmarklet is like a simple kind of browser extension.

Automating Assignees

I've created a free bookmarklet to streamline two scenarios:

Assigning work to only a specified list of students.
Assigning work to all students except a specified list.

For example, assigning an advanced math homework or a differentiated language assignment.

Let's say you have a group of five students – Scooby, Shaggy, Fred, Velma, and Daphne – that you sometimes assign work to. Your current workflow is something like:

Look up which students are in the group.
Open the drop-down list of assignees.
Uncheck "All students".
Check each of the five students.
Close the assignees list.

Use the bookmarklet generator to create a bookmarklet that automates these repetitive clicks. Your new workflow is as simple as:

Click the "Mystery Incorporated" bookmarklet.

If you have multiple groups, you can easily create a bookmarklet for each one.

Troubleshooting

If you see an error message when you run the bookmarklet, follow these steps:

Read the error message and look for troubleshooting tips. If you need to edit student names, open the bookmarklet generator and run your bookmarklet.
Changes to the Google Classroom interface may inadvertently break the bookmarklet. Open the bookmarklet generator and run your bookmarklet(s) to check for updates.

If all else fails, leave a comment below describing the issue and the error message you receive. I will assist as time allows.

Segment S3 uploads by Cognito user

Tue, 14 Feb 2017 00:00:00 GMT

Photo by Tim Evans on Unsplash.

Continuing my last post about combining AWS Cognito with EvaporateJS to upload to S3, we'll use S3 policies to isolate Cognito users into subdirectories of the S3 bucket.

AWS Policies to Isolate Users

Imagine you want User 1 to have exclusive access to a /Users/1/ subdirectory of your S3 bucket. You could update the S3 bucket's policy to specify that only User 1 has access to /Users/1/. Then what if you want User 2 to have exclusive access to /Users/2/? You could amend the bucket's policy to add rules for User 2. What about users 3, 4, and 5? Or the next 1,000 users of your site? Obviously this approach doesn't scale well.

Luckily AWS supports variables in policies. Instead of explicitly listing users, you could write a generic policy allowing User X to access /Users/X/.

Policy to Isolate Cognito Users

To use this approach with Cognito, I chose the cognito-identity.amazonaws.com:sub policy variable. This evaluates to the Cognito user's Identity ID, which follows the format region:guid. Here's an example policy.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AllowAccessToPrivateCognitoUserFolder",
			"Effect": "Allow",
			"Principal": {
				"AWS": ["arn:aws:iam::12345:role/Cognito_Auth_Role"]
			},
			"Action": [
				"s3:GetObject",
				"s3:PutObject",
				"s3:DeleteObject",
				"s3:AbortMultipartUpload"
			],
			"Resource": [
				"arn:aws:s3:::bucket-name/Users/${cognito-identity.amazonaws.com:sub}",
				"arn:aws:s3:::bucket-name/Users/${cognito-identity.amazonaws.com:sub}/*"
			]
		}
	]
}

The highlighted lines require additional attention:

On line 8, enter the role used by your authenticated Cognito users. This is listed in the Cognito area of the AWS Console, under Federated Identities. In the identity pool settings, click "Edit identity pool" in the top-right corner. The role's name is displayed in the "Authenticated role" menu. However, the bucket policy needs the role's full ARN, so you can hop over to the Roles list in IAM, find the role named in the identity pool's settings, and click it to find the Role ARN. Update line 8 with this value, wrapped in double quotes.
If you use Cognito's group feature to assign a custom IAM role to some users, your policy must include the ARN of every role that needs access to S3. Wrap each role ARN in double quotes and separate entries with commas. For example, if your "Moderators" group has a custom IAM role, you would list both roles like:
```
"Principal": {
    "AWS": [
        "arn:aws:iam::12345:role/Cognito_Auth_Role",        "arn:aws:iam::12345:role/Cognito_Moderator_Role"    ]
},
```
On lines 17 and 18, replace bucket-name with the name of your S3 bucket.

Code to Isolate Cognito Users

Thanks to the AWS policy above, User 2 will be unable to access /Users/1/. But how do we send a user's uploads to the correct subdirectory? Building on the S3Manager code in the previous article, I chose to pass the uploader's ID during instantiation.

const getS3Manager = () => {
	// skipping unchanged code...
	const {
		accessKeyId,
		identityId,		secretAccessKey,
		sessionToken,
	} = AWS.config.credentials;
	const s3Manager = new S3Manager(
		accessKeyId,
		secretAccessKey,
		sessionToken,
		identityId,	);
	// skipping unchanged code...
};

class S3Manager {
	constructor(accessKey, secretKey, sessionToken, userId) {
		// skipping unchanged code...
		this.userId = userId;	}

	upload(file, transferHeaders = {}, options = {}) {
		// skipping unchanged code...
		name: `Users/${this.userId}/encoded/upload/path.extension`, // TODO: Specify this		// skipping unchanged code..
	}
}

Now every Cognito user automatically has a private subdirectory in the S3 bucket that only they can access.

Calculate AWS v4 Signature with client-side JavaScript

Fri, 10 Feb 2017 00:00:00 GMT

Photo by Ray Reyes on Unsplash.

My team recently completed a challenging 5-month product build that integrates with many Amazon Web Services (AWS) features. We had the opportunity to use some new services, namely AWS Cognito, which forced us to pioneer new solutions more than a typical project. First up: calculating AWS v4 signatures client-side to integrate EvaporateJS with AWS Cognito.

Benefits of AWS Cognito

Although some parts of AWS Cognito feel incomplete (multi-factor authentication and documentation come to mind), it offers great potential by allowing users to authenticate directly to the AWS ecosystem. When Member Matt logs in, he can run client-side JavaScript to authenticate and talk directly with the AWS API. And when Admin Amber logs in, Cognito's group feature allows her to run elevated commands that Matt cannot. This lets us securely interact with AWS without intermediary servers to proxy commands.

Case in point: uploading private files to AWS S3. We want users of our product to store sensitive files in S3 without the ability to read or overwrite another user's files. We could upload files to a custom API that identifies the user and saves their files to S3, but this adds a lot of overhead. Instead, Cognito gives each user their own AWS credentials so they can read and write directly to S3, and we can write AWS policies to isolate users (see "Segment S3 uploads by Cognito user").

Uploading to S3 with EvaporateJS

To upload files to S3 we chose EvaporateJS, which immediately added value like multi-part uploads, pausing, resuming, and progress callbacks.

The following code uses EvaporateJS 1.x. The current version of EvaporateJS, 2.x, adds customAuthMethod so the implementation is slightly different. See this EvaporateJS 2.x example by paolavness.

import { assign, curry, mapKeys, map } from 'lodash';
import Evaporate from 'evaporate';

// These cryptography functions are defined later in the articleimport { hashSha256ToHex, hmacSha256 } from './crypto';
/**
 * Generate the signature for AWS to authenticate the request.
 * @param {String} secretKey - The IAM secret key of the AWS user making the request
 * @param {Object} response - The response from the signerUrl or awsLambda requests
 * @param {String} stringToSign - The "string to sign" generated in the previous step
 *   of the AWS Signature v4 process.
 * @returns {String} - The signature included in the Authorization header for AWS
 *   to authenticate the request.
 */
const signRequest = (secretKey, response, stringToSign) => {
	// stringToSign is URL-encoded by EvaporateJS, but the signature must use the decoded string.
	const stringToSignDecoded = decodeURIComponent(stringToSign);
	// Signing is defined later in the article};

class S3Manager {
	/**
	 * Create an instance of S3Manager.
	 * @constructor
	 * @param {String} accessKey - Uploader's AWS access key
	 * @param {String} secretKey - Uploader's AWS secret key
	 * @param {String} sessionToken - Uploader's AWS session token
	 */
	constructor(accessKey, secretKey, sessionToken) {
		this.manager = new Evaporate({
			aws_key: accessKey,
			awsRegion: 'my-aws-region-identifier', // TODO: Specify this
			awsSignatureVersion: '4', // this is default in EvaporateJS 2.x
			bucket: 'my-s3-bucket-name', // TODO: Specify this
			// Required for AWS Signature Version 4
			cryptoHexEncodedHash256: hashSha256ToHex,
			signResponseHandler: curry(signRequest)(secretKey),
		});
		this.sessionToken = sessionToken;
	}

	/**
	 * Upload a file to AWS S3.
	 * @param {File} file - The File object uploaded to the browser
	 * @param {Object} transferHeaders - Key-value pairs of headers to add to the AWS request, like
	 *   x-amz-server-side-encryption or x-amz-meta-* headers
	 * @param {Object} options - EvaporateJS properties to set/override for this upload
	 */
	upload(file, transferHeaders = {}, options = {}) {
		const authHeaders = { 'x-amz-security-token': this.sessionToken };
		const params = assign(
			{
				file,
				name: 'encoded/upload/path.extension', // TODO: Specify this
				xAmzHeadersAtInitiate: assign({}, authHeaders, transferHeaders),
				xAmzHeadersCommon: authHeaders,
			},
			options,
		);
		return this.manager.add(params);
	}
}

To create an instance of S3Manager using our logged-in cognitoUser:

const getS3Manager = () => {
	return new Promise((resolve, reject) =>
		cognitoUser.getSession((sessionErr, tokens) => {
			if (sessionErr) {
				return reject(sessionErr);
			}

			// getSession() will populate AWS.config.credentials
			AWS.config.credentials.get(credentialsErr => {
				if (credentialsErr) {
					return reject(credentialsErr);
				}

				const {
					accessKeyId,
					secretAccessKey,
					sessionToken,
				} = AWS.config.credentials;
				const s3Manager = new S3Manager(
					accessKeyId,
					secretAccessKey,
					sessionToken,
				);
				return resolve(s3Manager);
			});
		}),
	);
};

Signing the AWS Request

Using third-party libraries to interact with AWS requires us to cryptographically sign our own requests. The AWS SDK takes care of this but other libraries do not.

Signature Version 4 is the latest method for signing AWS requests. Of the four steps, EvaporateJS handles all except the third.

3. You use your AWS secret access key to derive a signing key, and then use that signing key and the string to sign to create a signature.

The EvaporateJS docs explain how to use AWS Lambda to sign requests. This is comparable to the Custom API proxy described earlier, in that we need to use our AWS Secret Key when calling AWS but we can't reveal it to the user. But with Cognito, that's no longer an issue! How can we use the Cognito user's credentials to sign the EvaporateJS request?

There are many popular client-side cryptography libraries, and I played with the v4 signature in several. Here are examples of the signRequest function from the code above:

These examples calculate the v4 signature of the stringToSign variable. To integrate with EvaporateJS, be sure to instead sign the stringToSignDecoded variable in the signRequest function.

Still More Cryptography

Finally, EvaporateJS uses other cryptographic operations, some required (like cryptoHexEncodedHash256) and some just encouraged (like cryptoMd5Method). Here are implementations of those functions, compared side-by-side for accuracy:

I hope these examples help you dive into exciting libraries like AWS Cognito and EvaporateJS!

Code Style: Accessing JavaScript Variables

Sun, 25 Oct 2015 00:00:00 GMT

Photo by Kelly Sikkema on Unsplash.

JavaScript's syntax allows for a lot of shortcuts when accessing variables. But shorter code can also be confusing. Which techniques are the shortest? Which are the easiest to read? A recent discussion with my co-workers led me to dive deeper.

A wise adage says, "code is read more than it is written." So my priorities are:

Readability
Simplicity
Performance

We want a pattern that is easily recognizable and predictable so our brains don't need to process it. When code is erratic I have to actually study it to find the purpose, or conversely I make quick assumptions that could be wrong.

Our task is to access a deeply nested property -- settings.server.port -- in a safe way lest any property is undefined, then assign it to the property of another object. If settings.server.port is falsy (null, undefined, false, 0, etc.), we'll default to 80. Let's see how various methods stack up.

1. `if` Statements

As a baseline, let's consider how verbose we could be.

if (settings && settings.server && settings.server.port) {
	foo.bar = settings.server.port;
} else {
	foo.bar = 80;
}

This approach is recognizable by programmers from any language. But, by the same token, we don't leverage any of JavaScript's convenience syntax.

Plus, if we are quickly skimming this code it's not obvious that foo.bar gets set on both code paths; the else statement might just as easily set hasPort = false or run handleMissingPort(). Nor is it obvious that the if criteria merely check the property tree for undefined values, instead of including a property like !useDefaultPort. This pattern opens the door for unexpected logic to hide inside.

2. Self-executing Anonymous Function

bar: (function() {
        if (settings && settings.server && settings.server.port)
            return settings.server.port;
        return 80;
    }()),

By wrapping our logic in a self-executing function, it now runs as a single statement. This allows us to use it in a large object initialization block, like:

var foo = {
	bar: 0,
	baz: true,
};

We see this all the time in JavaScript, so all methods moving forward should accommodate that syntax.

We can be sure that the result of the anonymous function is assigned to the bar property, but like example #1 we would need to inspect the contents of the function to find side effects like hasPort = false or handleMissingPort(), and extra conditions like !useDefaultPort.

3. Ternary Operator

bar: (settings && settings.server && settings.server.port) ?
    settings.server.port :
    80,

Another improvement. The ternary operator's tighter syntax helps minimize the logic since only one statement can run per code path. It's possible that someone could execute an anonymous function in one of the paths, but that giant block of code would hopefully catch our eye.

Nevertheless, this format still leaves lots of room for unexpected logic. The conditional statement might still contain weird conditions and either code path might call a function with side-effects (e.g. getDefaultPortAndEmailAdmins()).

4. Conditional Shorthand

bar: (settings && settings.server && settings.server.port) || 80,

This technique is much shorter because it uses JavaScript syntax that you won't find in every language.

By comparison, PHP is also a loosely-typed language. But its conditional operators always return a Boolean.

// Conditionals in PHP
$and = ('Cat' && 'Dog');
echo $and . ': ' . gettype($and);
$or = ('Cat' || 'Dog');
echo $or . ': ' . gettype($or);
// 1: boolean// 1: boolean

When I was new to JavaScript, the results of the same code surprised me.

// Conditionals in JavaScript
var and = 'Cat' && 'Dog';
console.log(and + ': ' + typeof and);
var or = 'Cat' || 'Dog';
console.log(or + ': ' + typeof or);
// Dog: string// Cat: string

This behavior allows for succinct code because we can test and access the property at once. But beware that anyone new to JavaScript might misinterpret your code and think bar will be true or 80.

Technically there's still room for unexpected logic, same as the techniques above. But at least there is overall less code, leaving fewer opportunities for surprises.

5. `try-catch` Block

bar: (function() {
        try { return settings.server.port; }
        catch() { return 80; }
    }()),

This solution came from my co-worker and is smart because it's straightforward. All programmers should recognize the purpose of a try-catch block, and we've eliminated all conditional statements that could hide unexpected logic. It can also grow easily to access a property nested 5 or 10 levels deep.

On the downside, the anonymous block plus the try-catch block add a lot of boilerplate code. Because this method is uncommon in JavaScript (in my experience) it could backfire and trigger a lot of cognition to deduce what errors are being caught. And lastly try-catch adds some performance overhead in JavaScript, so avoid this technique in core parts of your app.

6. Utility Function

The last approach we'll consider is hiding all the object traversal code inside a utility function. You can write your own or import a third-party option like selectn.

bar: selectn('server.port', settings) || 80,

This option is the most succinct, and leaves zero room for side-effects or unexpected logic. It requires a slight learning curve to figure out what is happening, so you might think twice if your project is open source with many transient readers. But anyone who frequently reads the code can get used to it quickly.

Personally I would prefer a wrapper function to reverse the order of parameters and improve readability, like getProp(settings, 'server.port') || 80. selectn makes the second parameter optional with a fallback to the global scope.

UPDATE: I finally realized that my favorite library, Lodash, includes this functionality and it's everything I could ask for. See the _.get() method.

bar: _.get(settings, 'server.port') || 80,

Conclusion

Code style is inherently subjective. But if you value readability you should use a consistent technique that the reader can skim and move on without much thought. For a light library or app I like the Conditional Shorthand technique. If you're working with many nested properties, especially on an internal project, then the Utility Function technique may add a lot of value to your codebase.

A Parser for Conditional Expressions

Tue, 07 Jul 2015 00:00:00 GMT

Photo by Yung Chang on Unsplash.

I've created a tool at https://www.wimpyprogrammer.com/conditional-parser/ for analyzing conditional expressions.

A while back my co-worker and I encountered a tricky conditional statement on a legacy C# project:

!HasDeadlinePassed
&& Active
&& (IsVerified.HasValue
  && (IsVerified.Value
    || (!IsVerified.Value
      && (AccountType != "CannotValidate"
        && !String.IsNullOrWhiteSpace(AccountType)
      )
    )
  ) || importID.HasValue
)

While troubleshooting that portion of the code, we needed to understand all of the circumstances in which the expression would evaluate to true. "Surely there's a tool that can break this down for us!" I thought, but couldn't find one and decided to build my own.

The Conditional Expression Parser will divide a conditional statement into its operators and conditions, then list the combinations that would cause the entire expression to evaluate to true. It recognizes AND (&&), OR (||), and XOR (^) operators and is smart enough to ignore strings ("a && b") and function calls (foo(a && b)).

The source code is available on GitHub.

Managing Puppet modules on Vagrant with Librarian-Puppet

Wed, 29 Apr 2015 00:00:00 GMT

Photo by Iñaki del Olmo on Unsplash.

When it comes to source code control I'm a purist: I avoid storing any derived files (ex. generated CSS) or third-party code. So when I started using PuPHPet to generate Vagrant and Puppet configurations, I was dismayed by the gigantic /puphpet/puppet/modules directory. 3,000 files totaling 5 MB?! There must be a better way.

Instead of keeping the Puppet dependencies in our repository, librarian-puppet lets us automatically load these modules during setup of the Vagrant box. In our Vagrantfile we add a new setup script:

config.vm.provision :shell, :path => 'puphpet/shell/install-ruby.sh'
config.vm.provision :shell, :path => 'puphpet/shell/install-puppet.sh'
config.vm.provision :shell, run: 'always' do |s|    s.path = 'puphpet/shell/install-puppet-modules.sh'    s.args = '/vagrant/puphpet/puppet'endconfig.vm.provision :shell, run: 'always' do |s|
    s.path = 'puphpet/shell/windows-path-limit-fix.sh'
    s.args = '/vagrant/node_modules'
end

The setup script is based on the bootstrap script from vagrant-puppet-librarian.

#!/bin/sh

# Temporary directory in which to install puppet modules.
# Avoids issues with writing to a VirtualBox shared folder.
TMP_FOLDER="/var/tmp/install-puppet-modules/"

install()
{
    # Skip install if the temp folder already exists
    if [ -d "$TMP_FOLDER" ]; then

        echo "Skipping puppet dependencies because the temporary installation folder already exists"

    else

        # Create the temp folder and populate it with the Puppetfile, Puppetfile.lock, etc.
        mkdir -p $TMP_FOLDER && cd $TMP_FOLDER
        cp -r $1/* ./

        # NB: librarian-puppet might need git installed. If it is not already installed
        # in your basebox, this will manually install it at this point using apt or yum
        GIT=/usr/bin/git
        APT_GET=/usr/bin/apt-get
        YUM=/usr/sbin/yum
        if [ ! -x $GIT ]; then
            if [ -x $YUM ]; then
                echo "Installing git via yum"
                yum -q -y install git-core
            elif [ -x $APT_GET ]; then
                echo "Installing git via apt-get"
                apt-get -q -y install git-core
            else
                echo "No package installer available. You may need to install git manually."
            fi
        fi

        if [ `gem query --local | grep librarian-puppet | wc -l` -eq 0 ]; then
          echo "Installing the librarian-puppet gem"
          gem install librarian-puppet
        fi

        echo "Installing puppet dependencies"
        librarian-puppet install

        # Copy the installed modules back to the project
        rsync -Lr --inplace $TMP_FOLDER/.librarian $1/
        rsync -Lr --inplace $TMP_FOLDER/modules $1/
        rsync -Lr --inplace $TMP_FOLDER/Puppetfile.lock $1/
    fi
}

if [ ! "$1" -o -z "$1" ]; then
    echo "You must specify a destination for the Puppet module installation (recommend an absolute path)"
elif [ ! -d "$1" ]; then
    echo "Puppet module installation destination $1 must already exist"
else
    install $1
fi

librarian-puppet reads from our Puppetfile -- also provided by PuPHPet -- for a list of all the Puppet dependencies to download. It behaves similar to the Windows path limit fix:

The modules install to a temporary directory because I had issues running librarian-puppet install inside the /vagrant directory, presumably because it's shared with Windows. After installation, the modules are copied to /vagrant/puphpet/puppet/modules so Vagrant's setup can continue as usual.
If the temp directory already exists during setup, the script assumes that the Puppet dependencies are ready and aborts.
Temporary files are stored in /var/tmp to persist during restarts, and the run: 'always' parameter in the Vagrantfile ensures that the script will fire during both vagrant reload and vagrant up.

Finally we add two entries to our .gitignore.

puphpet/puppet/.tmp/
puphpet/puppet/modules/

Now we can keep our repository lean while leveraging the power of third-party Puppet modules!

Updated - Vagrant on Windows path limit fix

Fri, 24 Apr 2015 00:00:00 GMT

Photo by Jono on Unsplash.

I previously shared my experience using PuPHPet to host a Linux VM on my Windows computer for MEAN stack development. Since then I've made some improvements to the startup script I use to overcome Windows's 260 character path limit, which npm and bower easily exceed.

Here's the latest script (and another nod to the GitHub discussion that started it):

#!/bin/bash

BASE_FOLDER="/var/tmp/windows-path-limit-fix/"

create()
{
    # Generate a unique but predictable directory by hashing the folder mount point
    # From http://stackoverflow.com/a/1602389
    name=`echo $1 | md5sum | cut -f1 -d" "`
    destination="$BASE_FOLDER$name"

    # Skip setup if the folder already exists
    if [ ! -d "$destination" ]; then

        echo "Creating mount destination: $destination"
        eval "mkdir -p $destination"
        eval "chown -R www-data:www-data $destination"
        eval "chmod 775 $destination"

    fi

    if [ -d "$1" ]; then

        echo "Transferring files from $1 to $destination"
        rsync -Lr --inplace $1/ $destination/

    else

        echo "Creating mount target: $1"
        eval "mkdir -p $1"

    fi

    echo "mounting $destination at $1"
    eval "mount -o bind $destination $1"

}

if [ ! "$1" -o -z "$1" ]; then
    echo "You must specify a folder mount point (recommend an absolute path)"
else
    create $1
fi

And the updated method for calling it:

config.vm.provision :shell, :path => 'puphpet/shell/install-ruby.sh'
config.vm.provision :shell, :path => 'puphpet/shell/install-puppet.sh'
config.vm.provision :shell, run: 'always' do |s|  s.path = 'puphpet/shell/windows-path-limit-fix.sh'  s.args = '/vagrant/node_modules'endconfig.vm.provision :shell, run: 'always' do |s|  s.path = 'puphpet/shell/windows-path-limit-fix.sh'  s.args = '/vagrant/.bower-tmp'end
config.vm.provision :puppet do |puppet|
  puppet.facter = {
    'ssh_username'     => "#{ssh_username}",
    'provisioner_type' => ENV['VAGRANT_DEFAULT_PROVIDER'],
    'vm_target_key'    => 'vagrantfile-local',
  }

While the old script got the job done, it had many inconveniences. The new version makes several improvements:

Temporary files are stored in /var/tmp instead of /tmp, meaning they persist after the VM is restarted.
Building on that, the temporary directory name is now a hash of the mount point instead of choosing a new name every time. This allows us to detect when the directory is already present then remount it.
The updated provisioner syntax in the Vagrantfile causes the script to run during vagrant reload in addition to the usual vagrant up.

These updates avoid repeated setup of the mounted directories. No more npm install after every reload!

The Statistics of War (the card game)

Wed, 24 Dec 2014 00:00:00 GMT

Photo by Jack Hamilton on Unsplash.

During Thanksgiving Break I played the card game War with my girlfriend and remembered how wonderfully mindless it is. Because there's no strategy or decision-making, it was feasible to write a program to play the game. Once I could simulate one game, I could simulate several and collect some statistics.

Evidently the rules of War have some gaps, including what to do with the cards you win. Do you place them on the bottom of your deck, in order, or do you shuffle and reuse them once your deck is empty? Why not program both methods and compare? Here is a plot of both approaches after 100,000 trials of each. The x-axis is the number of turns required to complete the game, and the y-axis is the frequency of that outcome across 100,000 trials.

The data proved more interesting than I expected.

Shuffling the winnings led to much faster games — a mean of 262, a mode of 84, and a max of 2,702 turns.
When the winnings aren't shuffled, the numbers are less clear. Many games were seemingly infinite so I added a cap of 5,000 turns per game. 11,837 (11.8%) of the games hit this limit (and do not appear on the graph). But plenty of games ended after 4,000+ turns, so we could expect many of these long games to be finite as well. If we ignore the capped games, the mode is 104 turns.
Both plots show a binary distribution of the results, as you can see in the higher and lower groupings of dots. It turns out that games are more likely to finish after an even number of turns. Each player starts with 26 cards, so if there are no wars and one player wins every hand, the game will last 26 turns. For every turn that player loses, they must win an additional hand to cancel the loss, so the game still requires an even number of turns. Only a war will break this trend because 4 cards change possession instead of 1. A game with an odd number of wars will take an odd number of turns to complete (ignoring wars with fewer than 3 cards, when one player's deck is low).

As for the program, here it is in JavaScript. The highlighted lines show the changes made between the Shuffle and No-Shuffle versions.

(function() {
	var NUM_TRIALS = 100000, // # games to simulate
		MAX_TURNS = 5000; // maximum # of turns to avoid infinite loops

	// From http://stackoverflow.com/a/2450976
	function shuffle(array) {
		var currentIndex = array.length,
			temporaryValue,
			randomIndex;

		// While there remain elements to shuffle...
		while (0 !== currentIndex) {
			// Pick a remaining element...
			randomIndex = Math.floor(Math.random() * currentIndex);
			currentIndex -= 1;
			// And swap it with the current element.
			temporaryValue = array[currentIndex];
			array[currentIndex] = array[randomIndex];
			array[randomIndex] = temporaryValue;
		}
		return array;
	}

	function Card(o) {
		var self = this;
		self.suit = o.suit;
		self.value = o.value;

		// Generate a string representation of the card
		self.toString = function() {
			var displayValue = self.value;
			switch (self.value) {
				case 11:
					displayValue = 'J';
					break;
				case 12:
					displayValue = 'Q';
					break;
				case 13:
					displayValue = 'K';
					break;
				case 14:
					displayValue = 'A';
					break;
			}
			return self.suit + displayValue;
		};

		// Return a negative number if compare is higher than this card.
		// Return a positive number if this card is higher than compare.
		// Return zero if the cards are equal.
		self.compare = function(/*Card*/ compare) {
			return self.value - compare.value;
		};

		return self;
	}

	function Player(o) {
		var self = this,
			hand = [],
			discard = [];

		self.name = o.name || 'Player';

		// Draw one card from the player's hand
		self.draw = function() {
			if (hand.length === 0) {
				// Version 1: The discard pile is shuffled before using				//hand = shuffle(discard).concat(hand);				// Version 2: The discard pile is used in the order that it was created				hand = discard.concat(hand);				discard = [];
			}
			return hand.pop();
		};

		// Add one or more cards to the player's discard pile
		self.discard = function(cards) {
			if (Array.isArray(cards)) {
				discard = cards.concat(discard);
			} else {
				discard.push(cards);
			}
		};

		// Count the total number of cards in this player's possession
		self.numCards = function() {
			return hand.length + discard.length;
		};

		// Does the user have any cards remaining?
		self.hasCards = function() {
			return self.numCards() > 0;
		};

		// Generate a string representation of the player
		self.toString = function() {
			return self.name + ' (' + self.numCards() + ' cards)';
		};

		return self;
	}

	function Game(o) {
		var self = this,
			deck = shuffle(o.deck);

		// The number of turns that have elapsed in this game
		self.numTurns = 0;

		// Simulate a single game
		self.play = function() {
			var player1 = new Player({ name: 'John' }),
				player2 = new Player({ name: 'Jane' });

			// Split the deck between the two players
			var iDeck;
			for (iDeck = 0; iDeck < deck.length; iDeck++) {
				if (iDeck % 2) {
					player1.discard(deck[iDeck]);
				} else {
					player2.discard(deck[iDeck]);
				}
			}

			var player1Card,
				player2Card,
				cardCompare,
				winnings,
				isWar,
				numWarCards,
				iWarCards;

			while (player1.hasCards() && player2.hasCards()) {
				self.numTurns++;

				// End this game if it's taking too long, to avoid potentially infinite loops
				if (self.numTurns > MAX_TURNS) {
					return;
				}

				player1Card = player1.draw();
				player2Card = player2.draw();
				winnings = [player1Card, player2Card];

				cardCompare = player1Card.compare(player2Card);

				if (cardCompare > 0) {
					// Player 1 wins this turn
					player1.discard(winnings);
				} else if (cardCompare < 0) {
					// Player 2 wins this turn
					player2.discard(winnings);
				} else {
					// War!
					do {
						// If a player is completely out of cards now, they lose the game
						if (!player1.hasCards()) {
							// Player 2 wins this game
							isWar = false;
							player2.discard(winnings);
						} else if (!player2.hasCards()) {
							// Player 1 wins this game
							isWar = false;
							player1.discard(winnings);
						} else {
							isWar = true;

							// If either player doesn't have enough cards for war, bet what they have
							numWarCards = Math.min(
								player1.numCards() - 1,
								player2.numCards() - 1,
								3,
							);

							for (iWarCards = 0; iWarCards < numWarCards; iWarCards++) {
								winnings.push(player1.draw());
								winnings.push(player2.draw());
							}

							player1Card = player1.draw();
							player2Card = player2.draw();
							winnings.push(player1Card);
							winnings.push(player2Card);

							cardCompare = player1Card.compare(player2Card);

							if (cardCompare > 0) {
								// Player 1 wins the war and this turn
								isWar = false;
								player1.discard(winnings);
							} else if (cardCompare < 0) {
								// Player 2 wins the war and this turn
								isWar = false;
								player2.discard(winnings);
							} else {
								// The card values are equal again, continue the war
							}
						}
					} while (isWar);
				}
			}

			// Note: this is erroneous if MAX_TURNS was reached
			self.winner = player1.hasCards() ? player1 : player2;
		};

		return self;
	}

	// Generate the master deck of all 52 cards.
	// It will be reused and shuffled in each Game instance.
	var suits = ['&spadesuit;', '&clubsuit;', '&heartsuit;', '&diamondsuit;'],
		deck = [];

	var iSuit, iValue;
	for (iSuit = 0; iSuit < suits.length; iSuit++) {
		// J = 11, Q = 12, K = 13, A = 14
		for (iValue = 2; iValue <= 14; iValue++) {
			deck.push(new Card({ suit: suits[iSuit], value: iValue }));
		}
	}

	// Run the desired number of trials.
	// Populate wins with the number of turns and frequency of that outcome.
	var iTrials,
		thisGame,
		wins = [];
	for (iTrials = 0; iTrials < NUM_TRIALS; iTrials++) {
		thisGame = new Game({ deck: deck });
		thisGame.play();

		if (!wins[thisGame.numTurns]) {
			wins[thisGame.numTurns] = 1;
		} else {
			wins[thisGame.numTurns]++;
		}
	}
})();

To close, enjoy the amazing uniqueness of a well-shuffled deck.

Using the MEAN stack on Windows with PuPHPet

Tue, 14 Oct 2014 00:00:00 GMT

Photo by Liliane Limpens on Unsplash.

Lately I've been playing with Vagrant and Puppet to create virtual machines for local MEAN stack development. I quickly learned that hosting a Linux VM on a Windows host is tricky. Let's look at the challenges and workarounds for them.

Virtual Machines with Puppet and Vagrant

But before we dive in, let's define some terms.

A virtual machine (VM) is an operating system (OS) that you host within your primary OS. For example, you might run a Linux VM within your Windows OS. To work with VMs I like VirtualBox.
Vagrant is a tool that automates the setup of virtual machines to the specifications you define in your Vagrantfile. Just like you version your application code, you can version the code for creating the application environment.
Puppet adds Ruby script support to Vagrant, allowing for much more powerful setup routines.

In the past couple years I've worked with PHP, C#.NET, and Node.js, and gradually my computer grew messy with all of the runtimes and dependencies. Using the tools above I can create clean, controlled development VMs and throw them away when I'm done. Or if my project takes off, I can deploy my Vagrant configuration to providers like Amazon, Digital Ocean, Rackspace.

PuPHPet to the Rescue

This all sounds great, but as a Linux-novice and a Ruby-nothing I was soon overwhelmed by the demands of creating and managing these environments. I eventually found PuPHPet (think Puppet + PHP) which handles most of the hard work. PuPHPet is a web GUI for generating the Puppet scripts to meet your needs. It's described nicely in this article by its creator and this Sitepoint article. PuPHPet is named for its original PHP focus, but is evolving to support more stacks like MEAN.

Hosting Linux VMs on Windows

Now that PuPHPet has created our Linux VM, there will be some problems running it within Windows. Vagrant will map our Windows project directory (containing the Vagrantfile) to /vagrant on the Linux VM, so the two OSes need to play nicely together.

Symbolic Links

In order for VirtualBox to support Linux symbolic links in the shared folder, we need to enable the VBoxInternal2/SharedFoldersEnableSymlinksCreate property. Conveniently this can be done through our PuPHPet configuration file.

provider:
  virtualbox:
    modifyvm:
      natdnshostresolver1: on
    setextradata:
      VBoxInternal2/SharedFoldersEnableSymlinksCreate/vagranthome: '1'
  vmware:
    numvcpus: 1
  parallels:
    cpus: 1

The vagranthome part of this setting is the name of the Vagrant shared folder. It seems to vary depending on which version of Vagrant you use. To verify the name, open the Shared Folders list in the VirtualBox VM properties and find the name of the Machine Folder that maps to the folder containing your Vagrantfile.

Windows pathname length limit

Windows file paths are limited to 260 characters, but tools like npm may easily create paths longer than this. The workaround I use is a mix of ideas from this GitHub issue because no single suggestion worked for me.

See Updated – Vagrant on Windows path limit fix for an updated version of these scripts.

#!/bin/bash

BASE_FOLDER="/tmp/windows-path-limit-fix/"

create()
{
    # Generate a unique, random directory
    name="$BASE_FOLDER$$"
    while [ -d "$name" ]
    do
        name="$BASE_FOLDER$$"
    done

    eval "mkdir -p $name"
    eval "chown -R www-data:www-data $name"
    eval "chmod 775 $name"
    eval "mkdir -p $1"

    eval "mount -o bind $name $1"
}

if [ ! "$1" -o -z "$1" ]; then
    echo "You must specify a folder mount point (recommend an absolute path)"
else
    create $1
fi

This script will create a temporary directory inside the VM and mount it at the specified path. Because the temporary directory is solely within Linux, it can fully support long paths. When either OS tries to access the mounted path, the operation will be transparently rerouted to the temporary directory.

Update your Vagrantfile to call this script for each directory you want to remap.

config.vm.provision :shell, :path => 'puphpet/shell/install-ruby.sh'
config.vm.provision :shell, :path => 'puphpet/shell/install-puppet.sh'
config.vm.provision :shell, :path => 'puphpet/shell/install-puppet-modules.sh'
config.vm.provision :shell, :path => 'puphpet/shell/windows-path-limit-fix.sh', :args => '/vagrant/node_modules'config.vm.provision :shell, :path => 'puphpet/shell/windows-path-limit-fix.sh', :args => '/vagrant/.bower-tmp'
config.vm.provision :puppet do |puppet|
    puppet.facter = {
        'ssh_username'     => "#{ssh_username}",
        'provisioner_type' => ENV['VAGRANT_DEFAULT_PROVIDER'],
        'vm_target_key'    => 'vagrantfile-local',
    }

With these problems wrapped up, we should have a fully-functioning Linux VM running within Windows.

Get bound property name in a KnockoutJS custom binding

Mon, 06 May 2013 00:00:00 GMT

Photo by Sai Kiran Anagani on Unsplash.

In a KnockoutJS custom binding, you may want to retrieve the name of the bound property. This usually isn't necessary; you can reference the valueAccessor parameter to get the bound data without knowing the property's name. But a recent project raised the need.

The custom binding that I wrote mirrors the syntax of the foreach binding, which accepts a few different formats. These are all valid:

<div data-bind="foreach: property">...</div>
<div data-bind="foreach: { data: property }">...</div>
<div data-bind="foreach: { data: property, option: value }">...</div>
<div data-bind="style: property2, foreach: property">...</div>

Let's look at a sample custom binding.

ko.bindingHandlers.myBinding = {
	init: function(
		element,
		valueAccessor,
		allBindingsAccessor,
		viewModel,
		bindingContext,
	) {
		// ...
	},
	update: function(
		element,
		valueAccessor,
		allBindingsAccessor,
		viewModel,
		bindingContext,
	) {
		// ...
	},
};

The bound data is passed as the valueAccessor parameter, but I couldn't find a reliable way to retrieve the name. So instead I parsed the properties of the bound DOM node, the element parameter.

var dataBind = element.attributes['data-bind'].nodeValue;
var propertyName = dataBind.match(
	/myBinding\s*:\s*(?:{.*,?\s*data\s*:\s*)?([^{},\s]+)/,);
return propertyName && propertyName.length
	? propertyName[1]
	: 'Value if the property name was not found';

In the dataBind.match, be sure to replace myBinding. This approach would return "property" in the examples above.

KnockoutJS custom binding for Quicksand

Sun, 28 Apr 2013 00:00:00 GMT

Photo by Alistair MacRobert on Unsplash.

This plugin has been moved into GitHub. And thanks to some updates to Quicksand, it now works with the latest versions of jQuery, Quicksand, and KnockoutJS.

Lately I've been experimenting with KnockoutJS, a JavaScript framework for streamlining client-side data and UI updates. I've found it wonderfully easy to implement Knockout's out-of-box functionality and obsolete tons of my custom code. This conversion was helped by Knockout's thorough documentation and numerous working examples. But I soon had the desire to integrate Knockout with Quicksand, a jQuery plugin for animating list sorts and filters. This quickly proved difficult, probably because I'm still new to Knockout, but with help from the Knockout forums I eventually created a custom binding to apply Quicksand animations to a Knockout observableArray. No one else has published such a binding so I'm sharing mine for others' benefit. But beware that it may not handle every circumstance you throw at it.

Overview

The quicksand custom binding mirrors Knockout's foreach (docs), which displays and tracks changes to an observableArray (docs). Simply put, the quicksand binding aims for the same functionality but with fancier animations. The foreach binding can also be added to a template (docs) binding for greater flexibility. So I've created a quicksandTemplate to mirror that functionality as well. The quicksandTemplate binding depends on functions in the quicksand binding.

Code

Here is the JavaScript code for the custom bindings, written against KnockoutJS 2.2.1.

ko.bindingHandlers.quicksand = {
	/**
	 * Create the hidden working space to facilitate the Quicksand animation.
	 *
	 * @param Element element  The DOM element with the KnockoutJS data bind.
	 * @param jQuery $repeatingElements  The HTML used to render each array element.
	 * @return jQuery  The Quicksand working space.
	 */
	createWorkspace: function(element, $repeatingElements) {
		// Generate a unique ID for the DOM element that will facilitate the Quicksand animation.
		var quicksandWorkspaceId = 'ko-quicksand-' + new Date().getTime();
		// Store the unique ID for reference during the "update" event.
		ko.utils.domData.set(
			element,
			'ko-quicksand-workspace-id',
			quicksandWorkspaceId,
		);

		// Look up the outermost element type in the rendering template and store it for reference during the "update" event.
		ko.utils.domData.set(
			element,
			'ko-quicksand-container-tag',
			$repeatingElements[0].tagName,
		);

		// Create the hidden DOM element that will facilitate the Quicksand animation.
		var $quicksandWorkspace = jQuery('<div/>', {
			id: quicksandWorkspaceId,
			style: 'display: none',
		});
		// Append it to the DOM for use during the "update" event.
		$quicksandWorkspace.appendTo('body');

		return $quicksandWorkspace;
	},
	/**
	 * Get the hidden working space used to facilitate the Quicksand animation.
	 *
	 * @param Element element  The DOM element with the KnockoutJS data bind.
	 * @return jQuery  The Quicksand working space.
	 */
	getWorkspace: function(element) {
		return jQuery(
			'#' + ko.utils.domData.get(element, 'ko-quicksand-workspace-id'),
		);
	},
	/**
	 * Get the name of the HTML tag that encompasses each array element.
	 *
	 * @param Element element  The DOM element with the KnockoutJS data bind.
	 * @return string  The container tag name.
	 */
	getContainerTag: function(element) {
		return ko.utils.domData.get(element, 'ko-quicksand-container-tag');
	},
	init: function(
		element,
		valueAccessor,
		allBindingsAccessor,
		viewModel,
		bindingContext,
	) {
		// Look up the provided template for rendering each element of the array.
		var $repeatingElements = jQuery(element).children();

		var $quicksandWorkspace = ko.bindingHandlers.quicksand.createWorkspace(
			element,
			$repeatingElements,
		);

		// Populate the workspace with the rendering template so it mirrors the contents of element.
		$repeatingElements.clone().appendTo($quicksandWorkspace);

		// Register the hidden DOM element for updates by Knockout.
		return ko.bindingHandlers['foreach']['init'](
			$quicksandWorkspace[0],
			valueAccessor,
			allBindingsAccessor,
			viewModel,
			bindingContext,
		);
	},
	update: function(
		element,
		valueAccessor,
		allBindingsAccessor,
		viewModel,
		bindingContext,
	) {
		// Unwrap the observable to subscribe to updates.  Check if the data property is implicit or explicit.
		ko.utils.unwrapObservable(
			valueAccessor().hasOwnProperty('data')
				? valueAccessor().data
				: valueAccessor(),
		);

		var $quicksandWorkspace = ko.bindingHandlers.quicksand.getWorkspace(
			element,
		);

		// Update the hidden DOM element using Knockout.
		ko.bindingHandlers['foreach']['update'](
			$quicksandWorkspace[0],
			valueAccessor,
			allBindingsAccessor,
			viewModel,
			bindingContext,
		);

		var quicksandOptions = valueAccessor().hasOwnProperty('options')
			? valueAccessor().options
			: {};

		// Run Quicksand to copy the contents of the hidden DOM element to the visible element using the Quicksand animation.
		jQuery(element).quicksand(
			$quicksandWorkspace.children(
				ko.bindingHandlers.quicksand.getContainerTag(element),
			),
			quicksandOptions,
		);
	},
};

ko.bindingHandlers.quicksandTemplate = {
	init: function(
		element,
		valueAccessor,
		allBindingsAccessor,
		viewModel,
		bindingContext,
	) {
		// Look up the provided template for rendering each element of the array.
		if (allBindingsAccessor().quicksandTemplate.hasOwnProperty('name')) {
			// The repeating template is defined in a named <script> block.
			var $repeatingElements = jQuery(
				jQuery.parseHTML(
					jQuery.trim(
						jQuery('#' + allBindingsAccessor().quicksandTemplate.name).html(),
					),
				),
			);
			var $quicksandWorkspace = ko.bindingHandlers.quicksand.createWorkspace(
				element,
				$repeatingElements,
			);
		} else {
			// The repeating template is the children of element.
			var $repeatingElements = jQuery(element).children();

			var $quicksandWorkspace = ko.bindingHandlers.quicksand.createWorkspace(
				element,
				$repeatingElements,
			);

			// Populate the workspace with the rendering template so it mirrors the contents of element.
			$repeatingElements.clone().appendTo($quicksandWorkspace);
		}

		// Register the hidden DOM element for updates by Knockout.
		return ko.bindingHandlers['template']['init'](
			$quicksandWorkspace[0],
			valueAccessor,
			allBindingsAccessor,
			viewModel,
			bindingContext,
		);
	},
	update: function(
		element,
		valueAccessor,
		allBindingsAccessor,
		viewModel,
		bindingContext,
	) {
		// Unwrap the observable to subscribe to updates.
		ko.utils.unwrapObservable(valueAccessor().foreach);

		var $quicksandWorkspace = ko.bindingHandlers.quicksand.getWorkspace(
			element,
		);

		// Update the hidden DOM element using Knockout.
		ko.bindingHandlers['template']['update'](
			$quicksandWorkspace[0],
			valueAccessor,
			allBindingsAccessor,
			viewModel,
			bindingContext,
		);

		var quicksandOptions = valueAccessor().hasOwnProperty('options')
			? valueAccessor().options
			: {};

		// Run Quicksand to copy the contents of the hidden DOM element to the visible element using the Quicksand animation.
		jQuery(element).quicksand(
			$quicksandWorkspace.children(
				ko.bindingHandlers.quicksand.getContainerTag(element),
			),
			quicksandOptions,
		);
	},
};

The logic above is actually pretty simple.

Create a hidden container that mirrors the content of the data-bound element.
Wire the hidden container, rather than the data-bound one, to be updated by Knockout.
Whenever the hidden container changes, synchronize its contents with the data-bound element using Quicksand.

Usage

The Quicksand custom bindings aim to support the same syntax as their built-in counterparts.

<ul data-bind="quicksand: arrayName">
	<li data-bind="text: elementPropertyName"></li>
</ul>

<ul data-bind="quicksand: { data: arrayName }">
	<li data-bind="text: elementPropertyName"></li>
</ul>

The optional options attribute can specify Quicksand parameters.

<ul
	data-bind="quicksand: { data: arrayName, options: { quicksandOption: value } }"
>
	<li data-bind="text: elementPropertyName"></li>
</ul>

<ul
	data-bind="quicksandTemplate: { foreach: arrayName, options: { quicksandOption: value } }"
>
	<li data-bind="text: elementPropertyName"></li>
</ul>

<ul
	data-bind="quicksandTemplate: { foreach: arrayName, name: 'template-id', options: { quicksandOption: value } }"
></ul>
<script type="text/html" id="template-id">
	<li data-bind="text: elementPropertyName"></li>
</script>

And to see the bindings in action, check out this jsFiddle based on Knockout's "animated transitions" example.

Microsoft Access is confused

Sun, 28 Apr 2013 00:00:00 GMT

Photo by Bill Aboudi on Unsplash.

The other day while working on a legacy Microsoft Access database I was reminded of how bizarre Access can be. When I ran the following query:

SELECT * FROM ([Authors]
INNER JOIN [Publications]
ON [Authors].id = Str([Publications].[authors].VALUE))
WHERE ( [Publications].[date] BETWEEN [Begin Date] AND [End Date] )

I received a peculiar error:

Microsoft Access The SQL statement could not be executed because it contains ambiguous outer joins. To force one of the joins to be performed first, create a separate query that performs the first join and then include that query in your SQL statement.

For a query with no outer joins, this was terribly unhelpful.

Researching the error, I found solutions like, "change all of the joins to INNER" which obviously didn't help me. So I began trying solutions that didn't make sense. Ultimately I found a version that worked:

SELECT * FROM ([Publications]
LEFT JOIN [Authors]
ON [Authors].id = Str([Publications].[authors].VALUE))
WHERE ( [Publications].[date] BETWEEN [Begin Date] AND [End Date] )

By reversing the order of the tables and changing the INNER join to a LEFT join, the query works. Neither change solved the problem alone.

So next time you're faced with a confusing Microsoft Access error message, try doing the opposite!

Relative paths in PHP INI

Thu, 11 Apr 2013 00:00:00 GMT

Photo by Jim Wilson on Unsplash.

I've always preferred to use relative file paths in my PHP INI, but ran into problems when running PHP from different directories. For example, a directive like error_log = "../logs/php_errors.log" will resolve to different paths when invoked by /public/index.php (e.g. Zend Framework) and /cli-config.php (e.g. Doctrine CLI). Some INI directives can be set at runtime by our application (see PHP_INI_ALL), but many cannot. I finally found the solution with environment variables.

In a past post, "Hosting PHP on IIS7", I detailed my typical server configuration for PHP sites. For maximum flexibility, every site has its own PHP INI. To point PHP to its INI file, I create a FastCGI environment variable named PHPRC. The PHPRC variable stores the absolute path of the directory containing php.ini. So the PHPRC variable contains a path like C:\inetpub\wwwroot\Blog\public.

Now I can simply use the PHPRC environment variable in my PHP INI. For example:

error_log = "${PHPRC}/../logs/php_errors.log"
upload_tmp_dir = "${PHPRC}/../temp/uploads"
session.save_path = "${PHPRC}/../temp/sessions"

These paths will automatically update depending on which server runs my code. Granted, these paths are not truly relative. But as long as every server contains an absolute path to the PHP INI, I may as well reuse it.

Microsoft Office Link Pre-fetching and Single Sign-On

Sun, 31 Mar 2013 00:00:00 GMT

Photo by Masaaki Komori on Unsplash.

I updated a site to use Single Sign-On (SSO) capability, specifically with Shibboleth, and was shocked to discover that links to the site in Microsoft Office files stopped working. This turns out to be a known issue with no definitive solution. So after thorough troubleshooting I created my own workaround, and as usual I hope sharing it will benefit others.

A Normal Shibboleth Sign-On

To understand the problem, let's first look at a successful Shibboleth sign-on. When a user visits a resource protected by Shibboleth, the Service Provider (SP) examines the request. For example, on IIS the SP is installed as an ISAPI filter in the protected website. The SP checks the user's cookies to verify that they have a valid session. If the session is valid, the request continues to the resource. If the user has an invalid or no session, they are redirected to the Identity Provider (IdP), the central authentication server. The original request details (the SP's address, the requested URL, etc.) are stored in a cookie by the IdP. If the user successfully authenticates to the IdP, they are transferred back to the SP, which is sent the request URL and the tokens of a valid session. Finally the SP redirects the user to the resource they requested. Phew!

Microsoft Office and Link Pre-fetching

Now we'll look at Microsoft Office and its peculiar handling of links. When you click a hyperlink in an Office file (Word document, Excel spreadsheet, PowerPoint presentation, etc.), the URL is not immediately passed to your browser. First, Office will internally fetch the address. If the link returns a 3xx Redirection code, Office will request the new address and repeat. If the link returns a 4xx Client Error code, Office will abort the request and tell the user that the link is unavailable, having never opened a browser (I'm guessing that's the whole point of this "feature"). And if the link returns a 2xx Success code, the URL is finally opened in a browser. Only the final URL is passed to the browser; any redirects are masked by Office.

Microsoft Office and SSO

Microsoft Office conflicts with Shibboleth (and other SSO systems, I expect) because of cookies. When Office requests the protected resource, the SP redirects to the IdP login page. Office follows the redirect and the IdP login page returns a 200 response, waiting for the user's credentials. Office opens the IdP login page address in the user's browser, but the cookies describing the request are lost. After the user authenticates, the IdP doesn't know how to return the user to their SP and displays an error. Thus hyperlinks in Office files never load.

Client-side Solutions?

Microsoft KB #899927 (see "Hyperlinks from Office to Internet Explorer or to another Web browser") describes some client-side workarounds for this problem that will alter Office's behavior. But the registry changes could have unexpected side effects. Depending on your audience, a registry change may be inconvenient or impossible.

A Server-side Solution

Ultimately we want a server-side solution that will transparently fix the problem for the user.

To start, we need finer control over Shibboleth's redirects. By default, the SP will automatically detect invalid sessions and redirect to the IdP. We need to disable this behavior and enable "lazy sessions" instead. In the <RequestMap> section of your SP configuration you'll find a <Host> section like this.

<Host name="example.com" authType="shibboleth" requireSession="true" applicationId="xxx">
    <Path name="favicon.ico" requireSession="false"/>
</Host>

Make note of any unprotected paths, like favicon.ico in the example above.

Update your SP's configuration to set requireSession="false".

<Host name="example.com" authType="shibboleth" requireSession="false" applicationId="xxx"/>

Now the site is unprotected, with nothing to require authentication. We will shift this responsibility to a URL rewriter. On IIS I use Ionic's ISAPI Rewrite.

RewriteCond  %{HTTP_SHIB_IDENTITY_PROVIDER} ^\s*$
RewriteCond  %{REQUEST_URI}  !/Shibboleth\.sso/ [I]
RewriteCond  %{REQUEST_URI}  !/favicon\.ico [I]
RedirectRule  .*  /Shibboleth.sso/Login?target=#X%{URL}#E

This rule will detect any unauthenticated requests and force them to log in. The /Shibboleth.sso/Login path will redirect the user from the SP to the IdP, and the target parameter will contain the URL-encoded address to which the user should return after authentication. Any unprotected paths like favicon.ico should be added as exceptions to this rule.

Now that Shibboleth's protection is restored, we can add a workaround for Microsoft Office. Before the rule above, add another one.

RewriteCond %{HTTP_USER_AGENT} ;\sms-office(\)|;) [I]
RewriteRule .* /Shibboleth-MSOffice.html [L]

This rule will check the user agent for ms-office (along with some syntax to minimize false positives). When Microsoft Office fetches a link, its user agent contains this unique ms-office string. The request from Office will be rewritten to a simple HTML file, causing the link to return a 200 response code and avoid the Shibboleth redirects.

The Shibboleth-MSOffice.html file must exist, but its contents are not critical. In theory, no person will see it. Still, I recommend some additional measures in case of false positives.

<html>
	<head>
		<title>Microsoft Office application detected</title>
		<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
		<meta http-equiv="refresh" content="0" />
	</head>
	<body>
		<p>
			The web server has detected that your visit originated from a Microsoft
			Office application.
		</p>
		<p>
			To make MS Office compatible with Single Sign-On (SSO) authentication, the
			visit has been halted.
		</p>
		<p>
			<strong>
				Please press F5 to refresh the browser and continue to your page.
			</strong>
		</p>
	</body>
</html>

This message will help users who are mistakenly caught by the rule, and the redirect will help in case the response is somehow cached.

With everything in place, the Office links will work again. When Office fetches a protected resource, the server's URL rewriter will detect the request and return an immediate 200 response. Office will pass the URL, unaltered, to the user's browser. When the user visits the address again, the request is validated by the SP and finally taken to the resource.

Multiple forms on the same page in Zend Framework

Thu, 21 Mar 2013 00:00:00 GMT

Photo by Christa Dodoo on Unsplash.

Processing two forms on the same page is not the simplest task in the Zend Framework. About a year ago I encountered this problem and studied others' approaches. Eventually I hacked together a solution with multiple actions and lots of redundant code. But I was recently making updates to this page and found a more elegant method for handling multiple forms.

Most solutions to this problem involve submitting each form to a different action. And many times that is appropriate. Imagine a login page with a search box in the corner. The login form can self-post and the search form can submit to its own action. If the search form fails validation, it can reappear with errors on its own page and the user won't miss the login form.

But sometimes your forms are similar enough that they should be processed by the same action. It's hard to describe such a scenario, but frankly it doesn't matter. If you're reading this post, you probably have such a need in your project and my example doesn't matter. So we'll imagine a user profile page with two forms for the user to update different parts of their account.

I've written previously about how I use forms across the MVC layers. If something in the code below doesn't make sense, please check my "MVC Form Layer" post.

First let's look at the view for the user profile edit page.

$this->editForm1->setMethod('post')
                ->setAction($this->url());

$submitForm1 = new Zend_Form_Element_Submit('submitForm1');$submitForm1->setLabel('Save Form 1');
$this->editForm1->addElement($submitForm1);

echo $this->editForm1;

$this->editForm2->setMethod('post')
                ->setAction($this->url());

$submitForm2 = new Zend_Form_Element_Submit('submitForm2');$submitForm2->setLabel('Save Form 2');
$this->editForm2->addElement($submitForm2);

echo $this->editForm2;

Notice that each form has a separate submit button, and each submit button has a different name in the form.

And now the edit profile action in the User module's Index controller.

public function editProfileAction()
{
    // TODO: Get user's ID from their session.
    //$userId = ???

    $userMapper = new Application_Model_Mapper_Users();
    $user = $userMapper->getById($userId);
    if($user === NULL) {
        // TODO: Handle failed profile lookup.
        return;
    }

    $this->view->editForm1 = $user->getForm('form1Name');
    $this->view->editForm2 = $user->getForm('form2Name');
    $request = $this->getRequest();
    if($request->isPost()) {
        $postData = $request->getPost();

        if(isset($postData['submitForm1'])) {            // The user submitted editForm1.
            if(!$user->saveForm1($this->view->editForm1, $postData)) {
                // Redisplay editForm1 with validation errors.
                // Repopulate the editForm2 from saved data.
                $this->view->editForm2->populateFromUser($user);
            } else {
                // TODO: Handle successful editForm1 submission.
                return;
            }
        } else { // OR elseif(isset($postData['submitForm2']))            // The user submitted editForm2.
            if(!$user->saveForm2($this->view->editForm2, $postData)) {
                // Redisplay editForm2 with validation errors.
                // Repopulate editForm1 from saved data.
                $this->view->editForm1->populateFromUser($user);
            } else {
                // TODO: Handle successful editForm2 submission.
                return;
            }
        }
    } else {
        // Populate both forms from saved data.
        $this->view->editForm1->populateFromUser($user);
        $this->view->editForm2->populateFromUser($user);
    }
}

Notice that we're checking the POST data for the names of the submit button that we created in the view. I've replaced some sections with // TODO comments to focus on the multi-form handling.

This is a bare bones example, but once you add more logic the advantages of a single action will stand out. If you submit each form to a different action, each action and view will share much of the same code. The approach above is much DRY-er.

Autoloading Symfony using Doctrine with Zend Framework

Sat, 16 Mar 2013 00:00:00 GMT

Photo by Dayne Topkin on Unsplash.

On some projects I use the Doctrine Object Relational Mapper, a fantastic library for modeling database objects. To combine it with Zend Framework 1, I use Guilherme Blanco's Bisna application resource. I recently upgraded a project to the latest version of Doctrine and suddenly couldn't autoload the Symphony classes. It took far too long for me to figure out the problem, so I hope I can spare someone else the same trouble.

Doctrine borrows some classes from the Symfony framework, namely for the Doctrine console and YAML parser. These classes retain the Symfony namespace, but reside in /Doctrine/Symfony. This adds a wrinkle to the autoloading process. If you use the Bisna application resource, you may have something like this in your application INI:

resources.doctrine.classLoader.loaderClass = "Doctrine\Common\ClassLoader"
resources.doctrine.classLoader.loaderFile = "Doctrine/Common/ClassLoader.php"

resources.doctrine.classLoader.loaders.symfony_console.namespace = "Symfony"
resources.doctrine.classLoader.loaders.symfony_console.includePath = "Doctrine"

However you choose to autoload, you need to specify that the root of the Symfony library is in the /Doctrine directory. Once the Bisna resource initializes during your application's bootstrap phase, any references to Symfony will autoload correctly.

The problem occurs when you use the Doctrine command line tool. This tool resides in Doctrine's /bin directory and includes the doctrine.php script. Depending on how you obtained Doctrine (PEAR, Composer, or a downloadable archive), a doctrine.php script may already be in the /bin directory. This doctrine.php script used to properly autoload Symfony, until a recent update. As you can see in the commit changelog, the call to \Doctrine\Common\ClassLoader was updated and no longer specifies /Doctrine as the base directory of the Symfony library. This code runs before your cli-config.php, and once Symfony's path is defined in the autoloading stack it's difficult to change. We need our own version of the doctrine.php script.

Luckily the Doctrine command line loads the script with a simple include('doctrine.php');. Because this path is relative, we can override the default with our own doctrine.php if we place it higher in the include_path. Our doctrine.php can mirror the default one, minus the autoloader settings.

<?php
// doctrine.php
$configFile = getcwd() . DIRECTORY_SEPARATOR . 'cli-config.php';

$helperSet = null;
if (file_exists($configFile)) {
    if ( ! is_readable($configFile)) {
        trigger_error(
            'Configuration file [' . $configFile . '] does not have read permission.', E_ERROR
        );
    }

    require $configFile;

    foreach ($GLOBALS as $helperSetCandidate) {
        if ($helperSetCandidate instanceof \Symfony\Component\Console\Helper\HelperSet) {
            $helperSet = $helperSetCandidate;
            break;
        }
    }
}

$helperSet = ($helperSet) ?: new \Symfony\Component\Console\Helper\HelperSet();

\Doctrine\ORM\Tools\Console\ConsoleRunner::run($helperSet);

This version will still kick off the cli-config.php script, which will bootstrap the application and configure the autoloader.

So in the end, the problem was complex but the solution is simple. Copy the /bin/doctrine.php file from the Doctrine library into your project and remove the \Doctrine\Common\ClassLoader setup code.

Don't host on a .255 IP address

Sun, 06 Jan 2013 00:00:00 GMT

Photo by Mathyas Kurmann on Unsplash.

Most of the websites that I build at work are for internal use, but last year I created a site with a public audience. Everything was going well post-launch until I received a handful of reports from users who couldn't access the site. They tried clearing cookies, switching browsers, switching computers, and resetting modems, but nothing worked. Visiting any page on the site would result in an error like

Internet Explorer cannot display the webpage

Unable to connect

They could view other sites hosted on my network, but nothing hosted on the same server as this public site. The PHP and server logs recorded nothing unusual. What was happening?

Lucky for me, this public site hosts content that our users really want to access. Otherwise countless people would have failed to reach the site but moved on without reporting it. For every user who contacted me, I would collect some basic information, ask them to attempt some tests, then apologize for the problems and suggest they use the Internet at somewhere like Starbucks.

After much frustration, I remembered that when the server was deployed months before, it was assigned an odd IP address. Its public IP address ended in .255. If you're familiar with networking, you recognize that an IP address ending in .255 is traditionally the broadcast address of a \24 network (#.#.#.0 - #.#.#.255). The broadcast address has special meaning within the network and is never assigned to any clients. But evidently this server is not on a \24 network because the .255 address worked fine, so I let it go.

I researched online and found some reports (but not many) to corroborate this problem with .255 IP addresses. Continuing with this hunch, the server farm administrators assigned an additional IP address to the server which didn't end in .0 or .255. I moved the website to the new IP address and, after the DNS change propagated, the problem was fixed. To date, no more users have reported problems accessing the site.

My testing suggests that the problem lies with the user's Internet Service Provider (ISP). Because a .255 IP address is normally significant, some ISPs may block outbound traffic to those addresses to prevent smurf attacks. While everyone agrees that this is a crude method, some ISPs clearly still use it.

If you don't want to ban a percentage of users from accessing your site, be sure you don't have public content hosted on a .255 IP address.

Preserve encoded slashes in URL

Fri, 14 Dec 2012 00:00:00 GMT

Photo by Kon Karampelas on Unsplash.

To use a reserved character in a URL without invoking its special meaning, the character must be URL encoded. For example, a ? separates the page name from the query string, a & separates query string parameters, and a = separates parameter names from their values. These characters must be converted to %3F, %26, and %3D respectively if we want them interpreted literally.

In MVC applications, these special characters are less common because URLs follow a pattern like /Module/Controller/Action/Parameter1/Parameter2. In this case, the most important reserved character is the forward slash. Let's see how to handle slashes in Zend Framework 1 URLs.

Encoding in a basic PHP application

A basic PHP-on-IIS application handles encoded characters well. Let's review the output of phpinfo().

Literal URL:	`http://localhost/index.php?param=value&%2F%3F%26%3D=%3D%2F%26%3F`
`_GET["param"]`	`value`
`_GET["/?&="]`	`=/&?`
`_SERVER["REQUEST_URI"]`	`/index.php?param=value&%2F%3F%26%3D=%3D%2F%26%3F`
`_SERVER["QUERY_STRING"]`	`param=value&%2F%3F%26%3D=%3D%2F%26%3F`

Note that the characters have been decoded in the $_GET superglobal, but the encoded version remains intact in the $_SERVER superglobal.

Encoding in an MVC PHP application

Compare this behavior to an MVC PHP application hosted on IIS.

Literal URL:	`http://localhost/param/value/%2F%3F%26%3D/%3D%2F%26%3F`
`_GET["?"]`	no value
`_SERVER["REQUEST_URI"]`	`/param/value/?&=/=/&?`
`_SERVER["QUERY_STRING"]`	`&=/=/&?`

This time the URL interpretation is all wrong. The encoded characters were decoded before IIS parsed the URL, so they were treated as query string delimiters. None of the PHP superglobals contain the encoded URL that appears in the browser bar. How can we pass encoded characters to the PHP MVC application?

Rewriting to the Rescue

The solution lies in our URL rewriter. On IIS, my tool of choice is Ionic's ISAPI Rewrite. To funnel MVC requests to the Zend Framework I use the following rule.

# IIRF.ini
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !\. [I]
RewriteRule ^[^?]*(\?.*|$) /index.php$1 [L]

These rules include some enhancements beyond the ones you might find online. Namely the RewriteRule will persist any query string parameters, which helps me use GET forms in my MVC application. But these customizations don't solve the encoding problem.

To preserve the encoded characters we need only add a modifier to the RewriteRule.

# IIRF.ini
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !\. [I]
RewriteRule ^[^?]*(\?.*|$) /index.php$1 [L,U]

The U modifier will save the original URL, before IIS decodes it, to the HTTP_X_REWRITE_URL server variable.

Literal URL:	`http://localhost/param/value/%2F%3F%26%3D/%3D%2F%26%3F`
`_SERVER["HTTP_X_REWRITE_URL"]`	`/param/value/%2F%3F%26%3D/%3D%2F%26%3F`

When the Zend Framework (Zend_Controller_Request_Http::setRequestUri()) parses the URL to determine the route and parameters, it will first look for the HTTP_X_REWRITE_URL variable containing the preserved URL. Voilà! Our MVC application sees our encoded characters.

MVC Form Layer

Sat, 01 Dec 2012 00:00:00 GMT

Photo by amin khorsand on Unsplash.

Forms are a tricky part of any Model-View-Controller (MVC) application. They incorporate validation, presentation, and security logic that spans all tiers of the application. Separating these concerns is difficult but important. I'll walk through how I use Zend_Form in Zend Framework 1, but the same principles should apply to any MVC application.

A User account form

For this article, imagine a basic form for users to create or update their account. To keep things simple, I've replaced some of the logic with // TODO: comments. Let's start with the form itself.

class Application_Form_Account
{
    const FIELD_USERNAME = 'username';

    public function init()
    {
        $this->setName('account');

        $username = new Zend_Form_Element_Text(self::FIELD_USERNAME);
        $username->setLabel('Username');
        // TODO: Add filters and validators

        $this->addElements(array($username));
    }

    public function populateFromUser(Application_Model_User $user)
    {
        $this->{self::FIELD_USERNAME}->setValue($user->getUsername());

        return $this;
    }
}

Because the form field names are referenced outside of this class, I assign them to class constants. The populateFromUser() function will receive a User object and copy its attributes to the appropriate form fields. Other populateFrom*() functions could be added to accommodate other objects.

The User model

Next we look at the User model. As a foundation I used the approach suggested by Matthew Weier O'Phinney, Zend Framework Project Lead, in "Using Zend_Form in Your Models". Here's the User model and its base class.

class Application_Model_User extends My_Model_Abstract
{
    protected $_formDefault = 'Account';

    public function saveFormAccount(Application_Form_Account $form, array $data)
    {
        if(!$form->isValid($data)) {
            return FALSE;
        }

        $this->setUsername($form->getValue($form::FIELD_USERNAME))
             ->save(); // Store the object; not discussed in this article

        return $this;
    }
}

abstract class My_Model_Abstract
{
    // The base class name or an array of class names to use when loading forms for this object.
    protected $_formBase = 'Application_Form_';

    // The name of the default form to load for this object.
    protected $_formDefault;

    /**
     * Retrieve an instance of Zend_Form that can be used to interact with this object.
     *
     * @param  string $type  The type of form to retrieve, if many are applicable.
     * @param  array|Zend_Config $options  The options to pass to the form.
     * @return Zend_Form
     */
    public function getForm($type = NULL, $options = array())
    {
        // Determine the name of the form to return.  This may be passed as a parameter
        // or set as a property of the model.
        if($type === NULL) {
            if(empty($this->_formDefault)) {
                throw new LogicException(sprintf('Default form not specified in %s::%s', __CLASS__ . '::' . __FUNCTION__);
            } else {
                $type = ucfirst($this->_formDefault);
            }
        } else {
            $type = ucfirst($type);
        }

        // Determine the full name of the form class.
        $class = '';
        if(empty($this->_formBase)) {
            throw new LogicException('Form base path(s) not specified in ' . __CLASS__ . '::' . __FUNCTION__);
        } else {
            foreach((array)$this->_formBase as $formBase) {
                if(class_exists($formBase . $type)) {
                    $class = $formBase . $type;
                    break;
                }
            }
        }

        if(empty($class)) {
            throw new LogicException('Unable to locate form "' . $type . '" in ' . __CLASS__ . '::' . __FUNCTION__);
        }

        return new $class($options);
    }
}

Compared to Matthew's example, I removed the $_forms property and added support for form options and multiple form bases.

The controller

Finally, we'll tie everything together in the controller.

class Application_IndexController extends Zend_Controller_Action
{
    public function indexAction()
    {
        $userId = $this->_getParam('userId', 0);
        $this->view->isNewUser = ($userId == 0);

        $userMapper = new Application_Model_Mapper_User(); // The User data mapper; not discussed in this article
        if($this->view->isNewUser) {
            $this->view->user = $userMapper->create(); // Custom mapper function to return a new Application_Model_User
        } else {
            $this->view->user = $userMapper->getById($userId);
        }

        $this->view->form = $this->view->user->getForm('account'); // Parameter is redundant
        if($this->_request->isPost()) {
            $postData = $this->_request->getPost();
            if(!$this->view->user->saveFormAccount($this->view->form, $postData)) {
                // Redisplay the form with validation errors.
            } else {
                // TODO: Display success message, 302-redirect user
            }
        } else {
            if($this->view->isNewUser) {
                // Display a blank form.
            } else {
                // Display the saved form values.
                $this->view->form->populateFromUser($this->view->user);
            }
        }
    }
}

The controller retrieves an instance of Application_Model_User and gets an Application_Form_Account from it. If loading the form for the first time, it is populated with the user's attributes. If the user submits the form, the form and its contents are passed to the User model to validate and save.

Displaying the form

To cap off this example, let's see the view that renders this action.

$this->form->setMethod('post')
           ->setAction($this->url());

$submit = new Zend_Form_Element_Submit('submit');

if($this->isNewUser) {
    $submit->setLabel('Register');
} else {
    $submit->setLabel('Update');
}

$this->form->addElement($submit);

// TODO: Set form and form element decorators

echo $this->form;

Presentation concerns like the submit button and form decorators are handled here because they aren't pertinent to the form-model interaction.

Conclusion

With this approach, all form operations are done in the appropriate tier. The form populates itself from models, the model digests all of the form data, and the view handles all of the rendering. This keeps the form flexible and reusable.

Using sprintf with multiple iterations

Sun, 25 Nov 2012 00:00:00 GMT

Photo by Jonathan Chng on Unsplash.

PHP's built-in sprintf() function is great for formatting complicated output and improving code readability. I recently had the need to output a string multiple times with some changes each time.

Basic output

To start, here's a basic usage of sprintf(). It prints a "Clear Date" hyperlink that uses jQuery to blank the field with an ID of "date".

$clearLink = <<<'NOWDOC'
<a onclick="%1$s('%2$s').val('')">%3$s</a>
NOWDOC;
echo sprintf($clearLink, '$j', //1
                         '#date', //2
                         'Clear Date'); //3

I highly recommend using the n$ position specifier syntax along with comments to number your sprintf() arguments. This makes it easier to change your format string, reuse arguments, and trace arguments to their placeholders.

This example is overly-simplified and the sprintf() is excessive, but it demonstrates the concept.

Advanced output with iterations

Now imagine that we need to print several clear links for various form inputs. Some of the sprintf() arguments will change but some will not. We can organize our output with multiple passes through sprintf().

$clearLink = <<<'NOWDOC'
<a onclick="%3$s('%1$s').val('')">%2$s</a>
NOWDOC;
// Fill placeholders that will not vary; preserve the rest
$clearLink = sprintf($clearLink, '%1$s', '%2$s',
                     '$j'); //3
// Fill the remaining placeholders
echo sprintf($clearLink, '#date', //1
                         'Clear Date'); //2
echo sprintf($clearLink, '#time', //1
                         'Clear Time'); //2
echo sprintf($clearLink, '#name', //1
                         'Clear Name'); //2

Notice that I've renumbered the placeholders in the original string. The placeholders that will vary on each iteration are numbered first. This avoids any renumbering on the first call to sprintf().

Of course this technique is not guaranteed to produce cleaner code. The added complexity may just obfuscate your logic. But keep it in your toolbox as an option for reusing complex templates.

Zend_Form_Decorator_Fieldset for individual elements

Fri, 23 Nov 2012 00:00:00 GMT

Photo by no one cares on Unsplash.

If you want to include a <fieldset> in your Zend_Form, you might try the Zend_Form_Decorator_Fieldset decorator. But it's not apparent that this decorator is meant for the form and not the individual elements in it. Instead, here's a custom decorator to render your Zend_Form_Element with a <fieldset>.

class My_Form_Decorator_Fieldset_Element extends Zend_Form_Decorator_Fieldset
{
    /**
     * Attribs that should be removed prior to rendering
     * @var array
     */
    public $stripAttribs = array(
        'maxlength',
        'jqueryparams',
        'size',
        'name',
        'readonly',
        'checked',
        'helper',
        'renderpassword',
        'onclick',
    );

    // Uncomment for Zend Framework versions < 1.12.0.
    /* public function getOptions()
    {
        $options = parent::getOptions();

        // Do not inherit the id of the element.  Ids must be unique.
        // See http://framework.zend.com/issues/browse/ZF-10803.
        if(array_key_exists('id', $options) &&
           $options['id'] == $this->getElement()->getId()) {
            unset($options['id']);
        }

        return $options;
    } */
}

I use this decorator to implement the form design at http://pea.rs/forms/top-labels.

Zend_Validate_Callback Success on False

Fri, 23 Nov 2012 00:00:00 GMT

Photo by Obi Onyeador on Unsplash.

What if you need to validate input using a callback function that will return False when the input is valid? Here's a custom validator to do the job.

class My_Validate_Callback_False extends Zend_Validate_Callback
{
    public function isValid($value, $context = NULL)
    {
        // Run the validator normally and collect the result.
        $result = parent::isValid($value, $context);

        if($result) {
            // The callback returned TRUE, so we should return FALSE.
            $this->_error(self::INVALID_VALUE);
            return FALSE;
        } elseif(in_array(self::INVALID_CALLBACK, $this->_errors)) {
            // The callback errored.  We should persist the FALSE result.
            return FALSE;
        } else {
            // The callback returned FALSE and did not error.  Clear the errors and return TRUE.
            $this->_errors = array();
            return TRUE;
        }
    }
}

When would you use this?

If you wrote the callback function, you control the result. But if your callback function is built into PHP or a library, it may not be so easy. You could wrap the function with a version that reverses the result, or you can use My_Validate_Callback_False.

For example, I have a user registration form with a CAPTCHA. New usernames are checked in the database for uniqueness. But I don't want to query the database unless the user has passed the CAPTCHA — this combats automated queries that probe for usernames. So the username validation chain includes a callback to Zend_Form_Element_Captcha->hasErrors().

$username = new Zend_Form_Element_Text('username');
$username->addPrefixPath('My_Validate', 'My/Validate/', 'validate')
         ->addValidator('Callback_False', TRUE, array(
               'callback' => array($captcha, 'hasErrors')));

Because hasErrors() will return False when the input is valid, I need My_Validate_Callback_False to reverse the normal behavior.

Zend_Validate_Callback with no context

Wed, 21 Nov 2012 00:00:00 GMT

Photo by Kelly Sikkema on Unsplash.

This class is super simple, so we'll begin with the code.

class My_Validate_Callback_NoContext extends Zend_Validate_Callback
{
    public function isValid($value)
    {
        return parent::isValid($value);
    }
}

How does it work?

During Zend_Form->isValid(), every validator on the form is passed an extra parameter called the "context". The context contains all of the data submitted with the form. This lets you validate one form field based on the value of another, like a Password Confirm field that must match a Password field.

One of the Zend Framework standard validators is Zend_Validate_Callback, which lets you validate the field using any function. For example, a Username field could validate with the following function:

public function isUsernameAvailable($username, $excludeId = FALSE)

The $excludeId parameter is supplied when the user has an existing account, and omitted for new user registrations. For example:

$userMapper = new Application_Model_Mapper_Users();
$username = new Zend_Form_Element_Text('username');
$username->addValidator('Callback', FALSE, array(
    'callback' => array($userMapper, 'isUsernameAvailable'),
    'options' => FALSE));

$userMapper = new Application_Model_Mapper_Users();
$username = new Zend_Form_Element_Text('username');
$username->addValidator('Callback', FALSE, array(
    'callback' => array($userMapper, 'isUsernameAvailable'),
    'options' => $this->_applicantId));

(In this case, you might also use Zend_Validate_Db_NoRecordExists, but I prefer not to leak so much model information into the form validation.)

What's the problem?

As it stands, the Register.php form above will trigger a Zend_Validate_Callback::INVALID_CALLBACK error:

An exception has been raised within the callback

That's because the form context will be appended as a parameter to the callback and interpreted as the $excludeId, fouling the database lookup.

Cue the custom validator. By overriding the Zend_Validate_Callback->isValid() function, we'll drop the form context before invoking the callback function. The registration form would be updated as follows:

$userMapper = new Application_Model_Mapper_Users();
$username = new Zend_Form_Element_Text('username');
$username->addPrefixPath('My_Validate', 'My/Validate/', 'validate')
    ->addValidator('Callback_NoContext', FALSE, array(
        'callback' => array($userMapper, 'isUsernameAvailable'),
        'options' => FALSE));

Now you can setup callback validators without worrying about the default parameters of the callback function.

Hosting PHP on IIS7

Tue, 20 Nov 2012 00:00:00 GMT

Photo by Peter Masełkowski on Unsplash.

Running PHP sites on Internet Information Services (IIS) is easy! And with a few extra steps, you can achieve a versatile and extensible setup to avoid management headaches down the line. Follow this guide to add PHP functionality to your IIS server and avoid common mistakes.

Install the PHP Binaries

Head over to http://windows.php.net/download/ and download the Zip of your desired PHP version. For IIS, get the "Non Thread Safe" version. If your desired version has an Installer download, ignore it and get the Zip instead. For this guide we'll get the latest version, 5.4.8.

Extract the Zip to a central location with a descriptive folder name, such as C:\Program Files (x86)\PHP\PHP_5_4_8_NTS_x86\. The folder name will distinguish these binaries from other versions you install later. For reasons discussed later, you should minimize spaces in the folder path.

Configure PHP in IIS

Verify that the CGI feature is enabled in IIS. In Administrative Tools > Server Manager enable Roles > Web Server > Application Development > CGI.

To help bridge the gap between development and server configuration, I recommend enabling Feature Delegation. In the IIS server properties, open Feature Delegation and change all of the properties to Read/Write. Or to enable Feature Delegation for a single site, choose "Custom Site Delegation" from the Actions menu. With read/write Feature Delegation, most changes made through the IIS Manager will be saved in the <system.webServer> section of the site's web.config, and changes made to the web.config will be enacted by IIS. This lets you track configuration changes in your source control and deploy them just like code updates.

Before continuing, let's discuss IIS's FastCGI settings. Every FastCGI module used in IIS — the executable and parameters — must be added to a whitelist. If you run a module not in the whitelist, you'll get an error like:

Error Summary

HTTP Error 500.0 - Internal Server Error <handler> scriptProcessor could not be found in <fastCGI> application configuration

Detailed Error Information

Module: FastCgiModule
Notification: ExecuteRequestHandler
Handler: PHP 5.4.8 NTS x86 via FastCGI
Error Code: 0x80070585

This whitelist improves security but complicates Feature Delegation. If you modify a handler in your web.config (like changing the PHP version), you must also update the IIS whitelist before the new handler will work. Every FastCGI module in the whitelist has its own configuration, including a distinct set of environment variables. To allow every site to have its own configuration, we'll create a separate entry for each one. This is more tedious than having one entry per PHP version, but is necessary to keep independent settings.

In the properties of the IIS site/application/folder/virtual directory for which you want to enable PHP scripts, open Handler Mappings. In the Actions menu, choose "Add Module Mapping". Enter the following settings:

Request path: *.php
Module: FastCgiModule
Executable: (Path to php-cgi.exe)|my.app=(Name for application)
Name: (Descriptive name of PHP version)

The executable is the most complex part, so let's break it down.

First specify the absolute path to php-cgi.exe that you extracted earlier from the Zip. If the path contains spaces, enclose it in quotes. But these quotes are lost when writing to the web.config and must be constantly replaced. To avoid headaches later, I recommend using the 8.3 filename for any folders with spaces.
Next add a vertical bar (|) to separate the executable from its arguments.
Finally supply a custom parameter that will distinguish this handler in the list of FastCGI modules. The name you assign is for your reference only.
Altogether, the executable looks like C:\PROGRA~2\PHP\PHP_5_4_8_NTS_x86\php-cgi.exe|my.app=Blog.

Open "Request Restrictions" and set the following options:

Invoke handler only if request is mapped to: File or folder
Specify the verbs to be handled: GET,POST
Specify the access required by the handler: Script

Choose Yes when prompted to add the module to the FastCGI whitelist.

Customize your PHP INI

Your site's PHP configuration needs updating to work on IIS. If you're starting a new application, you'll find sample php.ini files in the Zip you downloaded earlier. Copy the sample INI to your application files and rename it to php.ini. Verify the following settings:

cgi.fix_pathinfo = 1
cgi.force_redirect = 0
extension_dir = "ext"
fastcgi.impersonate = 0 - This is contrary to the Microsoft guides — I'll explain in the Permissions section.

Configure FastCGI Module

Next we need to configure the newly-created FastCGI module. There are two ways to change these settings in the IIS Manager, both in the server properties. The first method is FastCGI Settings, but this doesn't let you edit the executable path if you upgrade PHP. So I recommend using the Configuration Editor. Choose a section of system.webServer/fastCgi and edit the collection. Use the my.app argument to find the new FastCGI entry.

Set instanceMaxRequests to 10000 (Per Microsoft).
Add a PHP_FCGI_MAX_REQUESTS environment variable with a value of 10000 (Also per Microsoft).
Add a PHPRC environment variable with the absolute path of the directory containing php.ini. Ex. C:\inetpub\wwwroot\Blog\public
Any other site-specific variables should be added here, such as the APPLICATION_ENV environment variable used by the Zend Framework.

Installing PEAR

PEAR is a library of PHP extensions. If you're not using it now... don't start! Explore better options like Composer or Pyrus (PEAR2), and skip to the next section. But if you're supporting some legacy code that relies on PEAR, we'll install it now.

Download http://pear.php.net/go-pear.phar to your PHP installation directory, ex. C:\Program Files (x86)\PHP\PHP_5_4_8_NTS_x86\go-pear.phar.

We intentionally haven't added or changed any environment variables on the server, namely the PATH variable. This would tie our server to a single version of PHP and we want the flexibility to run several. Instead we'll use Windows batch scripts: any changes to the environment variable only occur in the scope of the script. Create the following batch script in your PHP installation directory.

@ECHO OFF
SET PHPFramework=%CD%

REM Append our PHP version to the PATH variable.
SET PATH=%PATH%;%PHPFramework%

REM When using PEAR from outside of its home directory, we need to specify PHP_PEAR_PHP_BIN as an absolute path to php.exe.
REM Use the same PHP installation that is specified in the PATH variable.
SET PHP_PEAR_PHP_BIN=%PHPFramework%\php.exe

REM Output the versions that will be used by this script.
REM After each "CALL" command we need to reset ECHO. The other batch files may reenable it.
CALL php -v
@ECHO OFF
ECHO;
CALL pear version
@ECHO OFF
ECHO;

ECHO Some PEAR package installers will verify that the prerequisite components are enabled in the PHP INI file. Because the PHP INI files on this server are per-site, not per-PHP version, these checks will fail. Verify that the prerequisites are enabled in the PHP INI of the site that will use the component, and then force the installation with "pear install -f ..."
ECHO;

REM Launch a shell so the user can issue commands.
CMD /T:17 /Q

Launch this batch script and run php go-pear.phar. When prompted to install PEAR system-wide or locally, enter local and confirm. All of the installation paths should auto-detect correctly, so press Enter to continue. When the installation completes, you can use the pear command to install the libraries you need. In your application's PHP INI, update the include_path to add the absolute path to the PEAR directory, ex. C:\Program Files (x86)\PHP\PHP_5_4_8_NTS_x86\PEAR.

Permissions

Lastly, let's configure the file permissions for your application. Your application may have special requirements, but these settings should get you started.

For some reason, PHP and Microsoft prescribe fastcgi.impersonate. I've asked around (1|2) for the rationale but can't get any answers. This seems to be a recommendation for IIS6, but thanks to improved security in IIS7 I advise disabling impersonation.

Here are the settings that I recommend:

Create a dedicated IIS application pool for your site. In the Advanced Settings, choose an Identity of Built-in account: ApplicationPoolIdentity. This option, introduced in IIS7, will create a new user account named IIS APPPOOL\[Application pool name].
In the IIS Authentication settings for your site, choose an Anonymous user identity of IUSR.
Update the NTFS permissions on your site's web directory to Grant read access and Deny write access to IUSR.
Update the NTFS permissions on your application directory and anything in your include_path to Grant read & list access to IIS APPPOOL\[Application pool name].
Update the NTFS permissions for any path where your application will write (such as upload_tmp_dir, session.save_path, and error_log) to Grant modify access to IIS APPPOOL\[Application pool name].

Conclusion

Congratulations, you've added PHP support to IIS! With this setup, you can host multiple PHP versions on the same server, customize environment variables and PHP settings per site, manage multiple copies of PEAR, and sandbox your sites with separate user accounts. Happy coding!

RSS Feed

Understanding Jest Mocks

1. Explicit unmock

2. Explicit mock

3. Core module

4. Automocking

5. Local module

6. Manual mock

Faster Google Classroom Assignments with a Bookmarklet

Browser Bookmarklets

Automating Assignees

Troubleshooting

Segment S3 uploads by Cognito user

AWS Policies to Isolate Users

Policy to Isolate Cognito Users

Code to Isolate Cognito Users

Calculate AWS v4 Signature with client-side JavaScript

Benefits of AWS Cognito

Uploading to S3 with EvaporateJS

Signing the AWS Request

Still More Cryptography

Code Style: Accessing JavaScript Variables

1. if Statements

2. Self-executing Anonymous Function

3. Ternary Operator

4. Conditional Shorthand

5. try-catch Block

6. Utility Function

Conclusion

A Parser for Conditional Expressions

Managing Puppet modules on Vagrant with Librarian-Puppet

Updated - Vagrant on Windows path limit fix

The Statistics of War (the card game)

Using the MEAN stack on Windows with PuPHPet

Virtual Machines with Puppet and Vagrant

PuPHPet to the Rescue

Hosting Linux VMs on Windows

Symbolic Links

Windows pathname length limit

Get bound property name in a KnockoutJS custom binding

KnockoutJS custom binding for Quicksand

Overview

Code

Usage

Microsoft Access is confused

Relative paths in PHP INI

Microsoft Office Link Pre-fetching and Single Sign-On

A Normal Shibboleth Sign-On

Microsoft Office and Link Pre-fetching

Microsoft Office and SSO

Client-side Solutions?

A Server-side Solution

Multiple forms on the same page in Zend Framework

Autoloading Symfony using Doctrine with Zend Framework

Don't host on a .255 IP address

Preserve encoded slashes in URL

Encoding in a basic PHP application

Encoding in an MVC PHP application

Rewriting to the Rescue

MVC Form Layer

A User account form

The User model

The controller

Displaying the form

Conclusion

Using sprintf with multiple iterations

Basic output

Advanced output with iterations

Zend_Form_Decorator_Fieldset for individual elements

Zend_Validate_Callback Success on False

When would you use this?

Zend_Validate_Callback with no context

How does it work?

What's the problem?

Hosting PHP on IIS7

Install the PHP Binaries

Configure PHP in IIS

Customize your PHP INI

Configure FastCGI Module

Installing PEAR

1. `if` Statements

5. `try-catch` Block