Evidently the aspnet_compiler.exe does not care about .refresh files (From my tests and what I read on the internets). It would be nice if it did as it would reduce the steps needed to build a WSP (Web Site Project) on a build server. I ended up just manually copying the assemblies into the bin folder before the build as follows in this nant script:

<mkiisdir dirpath="..\BabelFish.ServiceHost.Web" vdirname="BabelFish.ServiceHost.Web" />

<copy todir="..\BabelFish.ServiceHost.Web\bin">
    <fileset basedir="..\BabelFish.Services\bin">
        <include name="*.dll" />
    </fileset>
</copy>

<exec program="C:\WINDOWS\Microsoft.NET\Framework\${framework.version}\aspnet_compiler.exe" useruntimeengine="true">
    <arg value="-p" />
    <arg value="..\BabelFish.ServiceHost.Web" />
    <arg value="-v" />
    <arg value="BabelFish.ServiceHost.Web" />
</exec>

<deliisdir vdirname="BabelFish.ServiceHost.Web" />