Remote cp77tools and bugfixes (WolvenKit#301)

* Initial commit * Create README.md * better filenames * .gitignore apply * v0.1.0.1 - dump option - fixes * fix extraction and decompression * version + oodle * Update README.md * Update README.md * remove unneccesary files * .gitignore is now working * Hash finally found (WolvenKit#2) - Better memory management (don't load all file in ram in unpacking process) - Added optional extension for files - Some fields renamed - Added KARK detection - Some improvements on files extracting - Added SHA1Hash by Snazz#3248 * Multithreaded extracting + console progress bars * Filetime * Update README.md * Update README.md * extraction refactor * Update README.md * import tw3 filetypes * version upgrade * dump strings from archive * actions * derp * more dump options * more dump options. cleanup * Github actions (WolvenKit#4) * Github actions * Cleaned * multithreaded dump * Update README.md * whoops * gitattributes * Update OodleLZ.cs * Update README.md * Update OodleLZ.cs * System.Commandline - much better command line help and syntax * extraction output directory * extract with filenames * log * dump class layout * Update CR2WExport.cs - now also includes offset * more CLI options - more cli options and descriptions - archive refactor * write buffers instead of appending * better buffer extraction * cp77 cbitmap support * texture extraction * better extraction, texture types * fix build - version update * more hashes * fix cr2w info dumps * more hashes * update cr2w.dll from Wkit * better json dumps * version + readme * better texture extraction * fix typo * better texture support * add oodle tools * Create Multilayer_Mask.cs * fixes * update readme * fix dump task * mltemplate support - pretty json dump * Local Commit * Revert "Local Commit" This reverts commit ace88a2. * mltemplate mlsetup * Build Instructions (WolvenKit#16) * Create build.md * Changed capitalization in file name to fit better with naming conventions * dump class def option * quit * better hashing * Readme hint (with line ending noise :peeporun;) (WolvenKit#19) * add cp77 classes - remove tw3 classes * more classes * uncooking all * remove empty "solution items" filter * small refactor * speed up hash loading * switch progress bar library, the output is not as detailed but it fixes WolvenKit#21 * fix output directory logic * progressbar via ILoggerService * experimental kraken * wildcard and regex * Update archivehashes.csv * Multi-archive extraction (WolvenKit#24) * refactor * archive refactor * archive small stuff * Added outpath parameter to Rootcommands : CR2W & Oodle , if not given oupath = inputpath * Prevent crash if CR2W not given input path. Refactored Console.WriteLines * multiple input paths * gameuiCharacterCustomizationUiPresetValue - gameuiCharacterCustomizationUiPresetValue - some renames * fix a bug with cr2w header * OodleLZ : Using System was needed. ArchiveTask : Replaced C.writelines with new logging system. Program : Listen for Stringlogged Cr2wtask : New logging Dumptask : New logging Hashtask : New logging * packing * fix cr2w * Update Cr2wTask.cs * RGB, fixed progress bars * pack task * pack more * Added crc64 (WolvenKit#33) * add crc64 fix toc 3 tables * fix * fix buffer archiving * update version * Update README.md * Update README.md * Update Archive.cs - fix bug * Utility command for pulling latest Archive Hashes (WolvenKit#35) Co-authored-by: avail <[email protected]> * Fix (WolvenKit#36) * add crc64 fix toc 3 tables * fix * Fixed size in KARK Fixed size, zsize in table 2 Fixed offsetidx * packing bugfixes - fixed a bug where uncooking would crash the console - fixed a bug where the buffers would not get sorted by numeric value - fixed a bug where the filelist in a packed archive was not sorted by hash * Move Commands defintions into their own classes (WolvenKit#37) * Moved Command definitions into their own classes * Merged main * fix packing non-cr2w files * bugfixes - fixed a bug with the archiveManager - fixed a bug with generating missing hashes - fixed a bug with loading hashes * better hash handling * Update Program.cs - fixed a bug where the program would crash if no loose archive hashes were found * checks * refactoring IHashService - removed Maincontroller in favor of IHashService - refactored the IHashService and loading - fix archive naming * cr2wtask bugfix - fixed a bug where cr2w -a and -c would crash * some more enums * more enums * fix enums * multi-input commands, unit test begin cr2w fixes * csv uncooking * more hashes, csv uncooking * better hash service, logging - moved hashService to cp77.cr2w - fixed some UI issues with unpacking * Update CP77Tools.csproj - version update * Update AppSettingsService.cs fix path * List object inside buffersDict was not thread-safe, resulting in occasional null value insertion. Removed parallel execution. (WolvenKit#47) * add dumped classes * enums * fix missing classes * FNV 1A 64-bit hash algorithm optimizations (WolvenKit#49) * Streaming from zip-archive with hashes without extracting. (WolvenKit#50) * namespace changes WolvenKit.Cr2w -> CP77.Cr2w * remove unused legacy classes * Cross-platform support - removed memory-mapped files * Version update * correct enums * fix varname normalization * normalize class types * version updadte * fix class property * Added optional vertical flip when uncooking (WolvenKit#56) * Added optional vertical flip when uncooking * fixed typo and cleaned up texconwrapper * Merge branch 'feature-buffer-extraction' into dev - oodle decompression OSX support - extracting archives concatenates buffers instead of splitting - uncooking now extracts and decompresses buffers out of the cr2w - fixed a bug with buffer extraction * Update Archive.cs - better cross-platform handling * update name * update branch id * Revert "update name" This reverts commit cad7040. * Create nightly.yml * Delete nightly.yml * workflow dispatch * packing - added a rebuild option (to add split buffers back to a cr2w file) - refactored mod operations (packing, importing etc) - osx ooz instead of oodle * Update .gitignore * Update nightly.yml * Update nightly.yml * Update nightly.yml * Update nightly.yml * fix correct upload url * Update nightly.yml * Update README.md * Update README.md * unsafe dds rebuilding - added the option to unsafely pack textures as buffers * fix buffer rebuilding * better logging * Update README.md * Update README.md * bugfixes - fix packing bugs - fix rebuilding bugs - fix extraction bugs - refactor extraction - refactor uncooking - refactor decompression * Cr2w unit tests (WolvenKit#57) * Initial unit tests * Post dev merge * Fixed logging * Output as CSV, Better Exceptions * Flipped decompress flag * Pre-merge tidying up * Update README.md * some more enums * Implemented extracting from missing hash list (WolvenKit#59) * uncookTask * MLmask uncook support (WolvenKit#60) * Added mlmask uncook support * fixed interpolation issue resulting in pixelart * Update archivehashes.zip * refactor unbundling * add new commands, deprecate archive command * Update Uncook.cs - uncooking also extracts * fixed vertical flip (WolvenKit#62) * Update nightly.yml - create beta release instead of appending to existing release * Update dotnet-core.yml - don't create pre-release builds for main * add clean step before restore * fix restore bug * Update README.md * Uncook support for some extensions (WolvenKit#63) * Uncook support for cubemap and envprobe * Uncook support for texarray * fixed WolvenKit#65 * add export command * Version 1.1 (WolvenKit#67) * some more enums * Implemented extracting from missing hash list (WolvenKit#59) * uncookTask * MLmask uncook support (WolvenKit#60) * Added mlmask uncook support * fixed interpolation issue resulting in pixelart * Update archivehashes.zip * refactor unbundling * add new commands, deprecate archive command * Update Uncook.cs - uncooking also extracts * fixed vertical flip (WolvenKit#62) * Update nightly.yml - create beta release instead of appending to existing release * fix restore bug * version update * Update Uncook.cs - fixed a bug where uncooking would fail * Uncook support for some extensions (WolvenKit#63) * Uncook support for cubemap and envprobe * Uncook support for texarray * fixed WolvenKit#65 * add export command * version update Co-authored-by: Hambalkó Bence <[email protected]> Co-authored-by: Kay Hennig <[email protected]> * exclude reshaper stuff * rm remote projs * nuget specs * use nugets - use wolvenkit nugets - retarget to net5.0-windows * remove remote files * remove archive command * fix nuget dependencies * unit test base class * include unit tests * bugfixes - fixed a bug where uncooking would display an error despite it uncooking correctly - fixed some missing properties in classes - more bugfixes * Update WolvenKit.Cyberformats.csproj - update version * Update WolvenKit.Cyberformats.csproj - different version tag for nugets Co-authored-by: Eugene Pogrebnyak <[email protected]> Co-authored-by: R503B <[email protected]> Co-authored-by: Martin Winkelmann <[email protected]> Co-authored-by: Maxime Chambonnet <[email protected]> Co-authored-by: avail <[email protected]> Co-authored-by: snazk <[email protected]> Co-authored-by: phrisk <[email protected]> Co-authored-by: avail <[email protected]> Co-authored-by: Aaron Shoemaker <[email protected]> Co-authored-by: waitxd <[email protected]> Co-authored-by: Kay Hennig <[email protected]> Co-authored-by: Hambalkó Bence <[email protected]>
TarekJor · Jan 23, 2021 · 9d79fb4 · 9d79fb4
1 parent 593f283
commit 9d79fb4
Show file tree

Hide file tree

Showing 44 changed files with 3,023 additions and 44 deletions.
diff --git a/CP77.CR2W/Archive/Archive.cs b/CP77.CR2W/Archive/Archive.cs
@@ -21,7 +21,7 @@
 
 namespace CP77.CR2W.Archive
 {
-    public class Archive
+    public class Archive : IGameArchive
     {
         #region constructors
 
@@ -37,25 +37,27 @@ public Archive()
         /// <param name="path"></param>
         public Archive(string path)
         {
-            Filepath = path;
+            ArchiveAbsolutePath = path;
 
             ReadTables();
         }
 
         #endregion
 
         #region properties
+        public EArchiveType TypeName => EArchiveType.Archive;
+
         public ArHeader Header { get; set; }
         public ArTable Table { get; set; }
-        public string Filepath { get; set; }
+        public string ArchiveAbsolutePath { get; set; }
 
         [JsonIgnore]
         public Dictionary<ulong, ArchiveItem> Files => Table?.FileInfo;
 
         public int FileCount => Files?.Count ?? 0;
 
         [JsonIgnore]
-        public string Name => Path.GetFileName(Filepath);
+        public string Name => Path.GetFileName(ArchiveAbsolutePath);
         #endregion
 
         #region methods
@@ -78,7 +80,7 @@ private void ReadTables()
             //     _table = new ArTable(new BinaryReader(vs), this);
             // }
 
-            using var vs = new FileStream(Filepath, FileMode.Open, FileAccess.Read);
+            using var vs = new FileStream(ArchiveAbsolutePath, FileMode.Open, FileAccess.Read);
             Header = new ArHeader(new BinaryReader(vs));
             vs.Seek((long) Header.Tableoffset, SeekOrigin.Begin);
             Table = new ArTable(new BinaryReader(vs), this);
@@ -146,7 +148,7 @@ byte[] ExtractFile(OffsetEntry offsetentry, bool decompress)
                 using var ms = new MemoryStream();
                 using var bw = new BinaryWriter(ms);
 
-                using var stream = new FileStream(Filepath, FileMode.Open, FileAccess.Read);
+                using var stream = new FileStream(ArchiveAbsolutePath, FileMode.Open, FileAccess.Read);
                 using var binaryReader = new BinaryReader(stream);
                 binaryReader.BaseStream.Seek((long) offsetentry.Offset, SeekOrigin.Begin);
 
@@ -171,6 +173,8 @@ byte[] ExtractFile(OffsetEntry offsetentry, bool decompress)
 
 
         #endregion
+
+
     }
 
 

diff --git a/CP77.CR2W/Archive/ArchiveItem.cs b/CP77.CR2W/Archive/ArchiveItem.cs
@@ -27,19 +27,17 @@ public class ArchiveItem : IGameFile
         public string FileName => string.IsNullOrEmpty(_nameStr) ? $"{NameHash64}.bin" : _nameStr;
         public string Extension => Path.GetExtension(FileName);
 
-        public IGameArchive Archive { get => throw new NotImplementedException(); set => throw new NotImplementedException(); }
+        public IGameArchive Archive { get; set; }
         public string Name { get => throw new NotImplementedException(); set => throw new NotImplementedException(); }
         public uint Size { get => throw new NotImplementedException(); set => throw new NotImplementedException(); }
         public uint ZSize { get => throw new NotImplementedException(); set => throw new NotImplementedException(); }
         public long PageOffset { get => throw new NotImplementedException(); set => throw new NotImplementedException(); }
 
         public string CompressionType => throw new NotImplementedException();
 
-        private Archive _parentArchive;
-
-        public ArchiveItem(BinaryReader br, Archive parent)
+        public ArchiveItem(BinaryReader br, IGameArchive parent)
         {
-            _parentArchive = parent;
+            Archive = parent;
             var mainController = ServiceLocator.Default.ResolveType<IHashService>();
 
             Read(br, mainController);

diff --git a/CP77.CR2W/Archive/ArchiveManager.cs b/CP77.CR2W/Archive/ArchiveManager.cs
@@ -17,11 +17,12 @@ namespace CP77.CR2W.Archive
 {
     public class ArchiveManager : CyberArchiveManager
     {
-
-        public Dictionary<string, Archive> Archives { get; set; }
-
         public static string SerializationVersion = "1.1";
 
+        public ArchiveManager()
+        {
+        }
+
         public ArchiveManager(DirectoryInfo indir)
         {
             _parentDirectoryInfo = indir;
@@ -33,15 +34,10 @@ public ArchiveManager(DirectoryInfo indir)
             Reload(indir);
         }
 
-
-        public ArchiveManager()
-        {
-        }
-
         private DirectoryInfo _parentDirectoryInfo;
 
         #region properties
-
+        public Dictionary<string, Archive> Archives { get; set; }
         public Dictionary<ulong, List<ArchiveItem>> Files { get; }
         public Dictionary<string, List<ArchiveItem>> GroupedFiles =>
 

diff --git a/CP77.CR2W/Modkit/Pack.cs b/CP77.CR2W/Modkit/Pack.cs
@@ -37,7 +37,7 @@ public static Archive.Archive Pack(DirectoryInfo infolder, DirectoryInfo outpath
             var outfile = Path.Combine(outpath.FullName, $"basegame_{infolder.Name}.archive");
             var ar = new Archive.Archive
             {
-                Filepath = outfile,
+                ArchiveAbsolutePath = outfile,
                 Table = new ArTable()
             };
             using var fs = new FileStream(outfile, FileMode.Create);

diff --git a/CP77.CR2W/Modkit/Uncook.cs b/CP77.CR2W/Modkit/Uncook.cs
@@ -248,10 +248,11 @@ public static int Uncook(CR2WFile cr2w, List<byte[]> buffers, ECookedFileFormat
                         {
                             var di = new FileInfo(infile.FullName).Directory;
                             TexconvWrapper.Convert(di.FullName, $"{newpath}", uncookext);
+                            uncooksuccess = true;
                         }
                         catch (Exception e)
                         {
-                            // silent
+                            uncooksuccess = false;
                         }
                     }
 
@@ -275,12 +276,11 @@ public static int Uncook(CR2WFile cr2w, List<byte[]> buffers, ECookedFileFormat
                     using var nstream = new FileStream($"{newpath}", FileMode.Create, FileAccess.Write);
                     redcsv.ToCsvStream(nstream);
 
-
+                    uncooksuccess = true;
                     break;
                 }
                 case ECookedFileFormat.json:
                 {
-
                     break;
                 }
                 case ECookedFileFormat.mlmask:
@@ -301,8 +301,6 @@ public static int Uncook(CR2WFile cr2w, List<byte[]> buffers, ECookedFileFormat
                 }
                 case ECookedFileFormat.cubemap:
                 {
-
-
                     if (!(cr2w.Chunks.FirstOrDefault()?.data is CCubeTexture ctex) ||
                         !(cr2w.Chunks[1]?.data is rendRenderTextureBlobPC blob))
                         return -1;
@@ -351,8 +349,6 @@ public static int Uncook(CR2WFile cr2w, List<byte[]> buffers, ECookedFileFormat
                 }
                 case ECookedFileFormat.envprobe:
                 {
-
-
                     if (!(cr2w.Chunks.FirstOrDefault()?.data is CReflectionProbeDataResource probe) ||
                         !(cr2w.Chunks[1]?.data is rendRenderTextureBlobPC blob))
                         return -1;
@@ -391,8 +387,6 @@ public static int Uncook(CR2WFile cr2w, List<byte[]> buffers, ECookedFileFormat
                 }
                 case ECookedFileFormat.texarray:
                 {
-
-
                     if (!(cr2w.Chunks.FirstOrDefault()?.data is CTextureArray texa) ||
                         !(cr2w.Chunks[1]?.data is rendRenderTextureBlobPC blob))
                         return -1;

diff --git a/CP77.CR2W/Types/cp77/animAnimNode_Base.cs b/CP77.CR2W/Types/cp77/animAnimNode_Base.cs
@@ -9,6 +9,7 @@ namespace CP77.CR2W.Types
 	public class animAnimNode_Base : ISerializable
 	{
 		[Ordinal(0)]  [RED("id")] public CUInt32 Id { get; set; }
+		[Ordinal(1)]  [RED("poseInfoLogger")] public animPoseInfoLogger PoseInfoLogger { get; set; }
 
 		public animAnimNode_Base(CR2WFile cr2w, CVariable parent, string name) : base(cr2w, parent, name) { }
 	}

diff --git a/CP77.CR2W/Types/cp77/animAnimNode_MathExpressionFloat.cs b/CP77.CR2W/Types/cp77/animAnimNode_MathExpressionFloat.cs
@@ -9,6 +9,7 @@ namespace CP77.CR2W.Types
 	public class animAnimNode_MathExpressionFloat : animAnimNode_FloatValue
 	{
 		[Ordinal(0)]  [RED("expressionData")] public animMathExpressionNodeData ExpressionData { get; set; }
+		[Ordinal(1)]  [RED("expressionString")] public CString ExpressionString { get; set; }
 
 		public animAnimNode_MathExpressionFloat(CR2WFile cr2w, CVariable parent, string name) : base(cr2w, parent, name) { }
 	}

diff --git a/CP77.CR2W/Types/cp77/animAnimNode_OnePoseInput.cs b/CP77.CR2W/Types/cp77/animAnimNode_OnePoseInput.cs
@@ -9,6 +9,9 @@ namespace CP77.CR2W.Types
 	public class animAnimNode_OnePoseInput : animAnimNode_Base
 	{
 		[Ordinal(0)]  [RED("inputLink")] public animPoseLink InputLink { get; set; }
+		[Ordinal(1)]  [RED("VisMask", 0, 0)] public CArray<animTransformIndex> VisMask { get; set; }
+		[Ordinal(2)]  [RED("VisAxes", 0, 0)] public CBool VisAxes { get; set; }
+		[Ordinal(3)]  [RED("VisNames", 0, 0)] public CBool VisNames { get; set; }
 
 		public animAnimNode_OnePoseInput(CR2WFile cr2w, CVariable parent, string name) : base(cr2w, parent, name) { }
 	}

diff --git a/CP77.CR2W/Types/cp77/animAnimNode_StackTracksShrinker.cs b/CP77.CR2W/Types/cp77/animAnimNode_StackTracksShrinker.cs
@@ -9,6 +9,8 @@ namespace CP77.CR2W.Types
 	public class animAnimNode_StackTracksShrinker : animAnimNode_OnePoseInput
 	{
 		[Ordinal(0)]  [RED("tag")] public CName Tag { get; set; }
+		[Ordinal(1)]  [RED("extenderNodeId")] public CUInt32 ExtenderNodeId { get; set; }
+
 
 		public animAnimNode_StackTracksShrinker(CR2WFile cr2w, CVariable parent, string name) : base(cr2w, parent, name) { }
 	}

diff --git a/CP77.CR2W/WolvenKit.Cyberformats.csproj b/CP77.CR2W/WolvenKit.Cyberformats.csproj
@@ -2,6 +2,18 @@
 
 	<PropertyGroup>
 		<TargetFramework>net5.0-windows</TargetFramework>
+		<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
+		<Authors>Hambalkó Bence, Moritz Baron</Authors>
+		<Company>WolvenKit</Company>
+		<Description>File formats (Cyberpunk 2077) for the WolvenKit Mod Editor.</Description>
+		<PackageLicenseExpression>MIT</PackageLicenseExpression>
+		<PackageProjectUrl>https://github.com/WolvenKit/Wolven-kit</PackageProjectUrl>
+		<RepositoryType>git</RepositoryType>
+		<RepositoryUrl>https://github.com/WolvenKit/Wolven-kit</RepositoryUrl>
+		<PackageTags>wolvenkit, cyberpunk2077</PackageTags>
+		<AssemblyVersion>1.0.1.0</AssemblyVersion>
+		<FileVersion>1.0.1.0</FileVersion>
+		<Version>1.0.1</Version>
 	</PropertyGroup>
 
 	<ItemGroup>

diff --git a/CP77.MSTests/ArchiveTests.cs b/CP77.MSTests/ArchiveTests.cs
@@ -0,0 +1,51 @@
+using System;
+using System.Collections.Generic;
+using System.Configuration;
+using System.IO;
+using System.IO.MemoryMappedFiles;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using Catel.IoC;
+using WolvenKit.Common.Services;
+using CP77.CR2W;
+using CP77.CR2W.Archive;
+using CP77.CR2W.Resources;
+using Microsoft.Extensions.Configuration;
+using Microsoft.VisualStudio.TestTools.UnitTesting;
+using Newtonsoft.Json;
+using Newtonsoft.Json.Converters;
+using WolvenKit.Common;
+
+namespace CP77.MSTests
+{
+    [TestClass]
+    public class ArchiveTests : GameUnitTest
+    {
+        [TestMethod]
+        public void Test_Unbundle()
+        {
+
+        }
+
+        [TestMethod]
+        public void Test_Uncook()
+        {
+
+        }
+
+
+        private void test_archive(string extension = null)
+        {
+            var resultDir = Path.Combine(Environment.CurrentDirectory, TestResultsDirectory);
+            Directory.CreateDirectory(resultDir);
+
+            var success = true;
+
+            List<Archive> archives;
+
+
+
+        }
+    }
+}
diff --git a/CP77.MSTests/CP77.MSTests.csproj b/CP77.MSTests/CP77.MSTests.csproj
@@ -0,0 +1,38 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net5.0-windows</TargetFramework>
+
+    <IsPackable>false</IsPackable>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="1.1.0" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />
+    <PackageReference Include="MSTest.TestAdapter" Version="2.1.1" />
+    <PackageReference Include="MSTest.TestFramework" Version="2.1.1" />
+    <PackageReference Include="coverlet.collector" Version="1.3.0" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <None Remove="appsettings.json" />
+    <Content Include="appsettings.json">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </Content>
+  </ItemGroup>
+
+  <ItemGroup>
+    <ProjectReference Include="..\CP77.CR2W\WolvenKit.Cyberformats.csproj" />
+    <ProjectReference Include="..\WolvenKit.Common\WolvenKit.Common.csproj" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <Content Update="C:\Users\ghost\.nuget\packages\wolvenkit.common\1.0.0\contentFiles\any\net5.0-windows7.0\DDS\texconv.exe">
+      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
+    </Content>
+    <Content Update="C:\Users\ghost\.nuget\packages\wolvenkit.cyberformats\1.0.0\contentFiles\any\net5.0-windows7.0\Resources\archivehashes.zip">
+      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
+    </Content>
+  </ItemGroup>
+
+</Project>