Compare commits
7 Commits
| Author | SHA1 | Date |
|---|---|---|
 Andrew Lalis
|
5b06f30df9 | |
|
Andrew Lalis
|
e33b594a5e | |
|
Andrew Lalis
|
540616db42 | |
|
Andrew Lalis
|
76099ddc99 | |
|
Andrew Lalis
|
99b3043686 | |
|
Andrew Lalis
|
4df5d82c08 | |
|
Andrew Lalis
|
4f0848c2d1 |
|
|
@ -0,0 +1,5 @@
|
|||
.distribugit_tmp/
|
||||
.distribugit_test_tmp/
|
||||
.idea/
|
||||
target/
|
||||
*.iml
|
||||
21
LICENSE
21
LICENSE
|
|
@ -1,21 +0,0 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2022 Andrew Lalis
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
36
README.md
36
README.md
|
|
@ -1,42 +1,34 @@
|
|||
# DistribuGit
|
||||
Tool for performing operations on many git repositories at once. It works by cloning a set of repositories, and applying an action to each repository.
|
||||
|
||||
The easiest way to use this toolset for automating git operations is to include it as a dependency in your Java project. You can get this project from [jitpack.io](https://jitpack.io/#andrewlalis/distribugit).
|
||||
The simplest way to get started is to get the [latest release](https://github.com/andrewlalis/distribugit/releases) JAR file, and run it in the command line.
|
||||
```
|
||||
java -jar distribugit-1.2.0.jar -t abc -s org-repo-prefix:concord/Java_ -a "ls"
|
||||
```
|
||||
The above command simply gets all repositories from the `corcord` GitHub organization which begin with `"Java_"`, and executes the `ls` command in each repository.
|
||||
|
||||
Here's an example of how one might use DistribuGit to apply a bash script to a set of repositories:
|
||||
In addition to the CLI, you can include DistribuGit as a dependency in your Java project. You can get this project from [jitpack.io](https://jitpack.io/#andrewlalis/distribugit).
|
||||
|
||||
Here's an example of how one might use DistribuGit to run `mvn test` on many repositories:
|
||||
|
||||
```java
|
||||
public static void main(String[] args) throws IOException {
|
||||
new DistribuGit.Builder()
|
||||
.selector(RepositorySelector.from(
|
||||
"https://github.com/andrewlalis/RandomHotbar.git",
|
||||
"https://github.com/andrewlalis/CoyoteCredit.git",
|
||||
"https://github.com/andrewlalis/SignalsAndSystems2021.git"
|
||||
"https://github.com/andrewlalis/CoyoteCredit.git"
|
||||
))
|
||||
.credentials(GitCredentials.ofUsernamePassword("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", ""))
|
||||
.action(RepositoryAction.ofCommand("/bin/bash", "../../test.sh"))
|
||||
.statusListener(new StatusListener() {
|
||||
@Override
|
||||
public void progressUpdated(float percentage) {
|
||||
System.out.printf("Progress: %.1f%%%n", percentage);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void messageReceived(String message) {
|
||||
System.out.println("Message: " + message);
|
||||
}
|
||||
})
|
||||
.strictFail(false)
|
||||
.cleanup(false)
|
||||
.build().doActions();
|
||||
.action(RepositoryAction.ofCommand("mvn", "test"))
|
||||
.build()
|
||||
.doActions();
|
||||
}
|
||||
```
|
||||
|
||||
In short, we need to specify the following things in order to use the tool:
|
||||
|
||||
- The `RepositorySelector` provides a list of URIs that we can use to clone the repositories. Make sure that the URI format matches the type of credentials you plan to use, i.e. HTTPS should begin with `https://`, and SSH should begin with `git@`. This component is **mandatory**.
|
||||
- The `RepositoryAction` is the action you want to perform on each repository. The `ofCommand` method can be used as a convenience to execute a command for each repository. Note that the script is executed within the repository's directory, which itself is in DistribuGit's working directory. Hence the `../../` in our example to point to a script in the directory from which our main method was started. This component is **mandatory**.
|
||||
- The `GitCredentials` are used to provide credentials in case you're trying to perform actions that only an authenticated user can. If you're using a GitHub personal access token, you can provide it like in the example above: by providing it as a username, with no password. By default, no credentials are provided.
|
||||
- The `RepositoryAction` is the action you want to perform on each repository. The `ofCommand` method can be used as a convenience to execute a command for each repository. Note that the script is executed within the repository's directory, which itself is in DistribuGit's working directory. Hence, use `../../` to point to a script in the directory from your main method was started. This component is **mandatory**.
|
||||
- The `GitCredentials` are used to provide credentials in case you're trying to perform actions that only an authenticated user can. If you're using a GitHub personal access token, you can provide it as a username, with no password. By default, no credentials are provided.
|
||||
- The `StatusListener` is a component that's used to send log output and occasional progress updates as DistribuGit runs. By default, this just outputs information to `System.out`.
|
||||
- `workingDir` denotes the directory in which DistribuGit will run. This directory may be completely deleted. By default, it is set to `.distribugit_tmp` in the current directory.
|
||||
- `strictFail` determines if we should quit as soon as any error occurs. If this is false, then we will continue the operations even if some repositories encounter errors. By default, this is set to **true**.
|
||||
|
|
|
|||
50
pom.xml
50
pom.xml
|
|
@ -6,7 +6,7 @@
|
|||
|
||||
<groupId>nl.andrewl</groupId>
|
||||
<artifactId>distribugit</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<version>1.3.1</version>
|
||||
|
||||
<properties>
|
||||
<maven.compiler.source>17</maven.compiler.source>
|
||||
|
|
@ -34,5 +34,53 @@
|
|||
<version>1.7.36</version>
|
||||
<scope>runtime</scope>
|
||||
</dependency>
|
||||
<!-- https://mvnrepository.com/artifact/org.kohsuke/github-api -->
|
||||
<dependency>
|
||||
<groupId>org.kohsuke</groupId>
|
||||
<artifactId>github-api</artifactId>
|
||||
<version>1.306</version>
|
||||
</dependency>
|
||||
<!-- https://mvnrepository.com/artifact/info.picocli/picocli -->
|
||||
<dependency>
|
||||
<groupId>info.picocli</groupId>
|
||||
<artifactId>picocli</artifactId>
|
||||
<version>4.6.3</version>
|
||||
</dependency>
|
||||
|
||||
<!-- https://mvnrepository.com/artifact/org.junit.jupiter/junit-jupiter-api -->
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-api</artifactId>
|
||||
<version>5.8.2</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<!-- https://mvnrepository.com/artifact/org.junit.jupiter/junit-jupiter-engine -->
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-engine</artifactId>
|
||||
<version>5.8.2</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-assembly-plugin</artifactId>
|
||||
<version>3.3.0</version>
|
||||
<configuration>
|
||||
<archive>
|
||||
<manifest>
|
||||
<mainClass>nl.andrewl.distribugit.cli.DistribuGitCommand</mainClass>
|
||||
</manifest>
|
||||
</archive>
|
||||
<descriptorRefs>
|
||||
<descriptorRef>jar-with-dependencies</descriptorRef>
|
||||
</descriptorRefs>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
</project>
|
||||
|
|
@ -9,10 +9,38 @@ import java.nio.file.Path;
|
|||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.ForkJoinPool;
|
||||
|
||||
/**
|
||||
* A single DistribuGit instance is used to execute a single sequence of
|
||||
* operations on a set of git repositories, as configured according to its
|
||||
* various components.
|
||||
* <p>
|
||||
* A DistribuGit object, when invoking {@link DistribuGit#doActions()} or
|
||||
* {@link DistribuGit#doActionsAsync()}, will perform the following series
|
||||
* of operations:
|
||||
* </p>
|
||||
* <ol>
|
||||
* <li>Call {@link RepositorySelector#getURIs()} to collect the list of
|
||||
* repositories that it will operate on.</li>
|
||||
* <li>Download each repository to the working directory.</li>
|
||||
* <li>Applies the configured {@link RepositoryAction} to all of the
|
||||
* repositories.</li>
|
||||
* <li>If provided, applies the configured finalization action to all of
|
||||
* the repositories.</li>
|
||||
* <li>If needed, all repositories are deleted.</li>
|
||||
* </ol>
|
||||
* <p>
|
||||
* Note that repositories are not guaranteed to be processed in any
|
||||
* particular order.
|
||||
* </p>
|
||||
*/
|
||||
public class DistribuGit {
|
||||
private final RepositorySelector selector;
|
||||
private final RepositoryAction action;
|
||||
private final RepositoryAction finalizationAction;
|
||||
private final GitCredentials credentials;
|
||||
private final StatusListener statusListener;
|
||||
private final Path workingDir;
|
||||
|
|
@ -22,9 +50,27 @@ public class DistribuGit {
|
|||
private int stepsComplete;
|
||||
private int stepsTotal;
|
||||
|
||||
/**
|
||||
* Constructs a DistribuGit instance.
|
||||
* @param selector A selector that provides a list of repository URIs.
|
||||
* @param action An action to do for each repository.
|
||||
* @param finalizationAction A final action to do for each repository,
|
||||
* after all normal actions are done.
|
||||
* @param credentials The credentials to use to operate on repositories.
|
||||
* @param statusListener A listener that can be used to get information
|
||||
* about the progress of the operations, and any
|
||||
* messages that are emitted.
|
||||
* @param workingDir The directory in which to do all git operations.
|
||||
* @param strictFail Whether to fail instantly if any error occurs. If set
|
||||
* to false, the program will continue even if actions
|
||||
* fail for some repositories.
|
||||
* @param cleanup Whether to perform cleanup after everything is done. This
|
||||
* will remove the working directory once we're done.
|
||||
*/
|
||||
public DistribuGit(
|
||||
RepositorySelector selector,
|
||||
RepositoryAction action,
|
||||
RepositoryAction finalizationAction,
|
||||
GitCredentials credentials,
|
||||
StatusListener statusListener,
|
||||
Path workingDir,
|
||||
|
|
@ -33,6 +79,7 @@ public class DistribuGit {
|
|||
) {
|
||||
this.selector = selector;
|
||||
this.action = action;
|
||||
this.finalizationAction = finalizationAction;
|
||||
this.credentials = credentials;
|
||||
this.statusListener = statusListener;
|
||||
this.workingDir = workingDir;
|
||||
|
|
@ -40,16 +87,41 @@ public class DistribuGit {
|
|||
this.cleanup = cleanup;
|
||||
}
|
||||
|
||||
public void doActions() throws IOException {
|
||||
/**
|
||||
* Performs the configured actions on the selected git repositories.
|
||||
* @throws IOException If an error occurs that requires us to quit early.
|
||||
* This is only thrown if {@link DistribuGit#strictFail} is true.
|
||||
*/
|
||||
public synchronized void doActions() throws IOException {
|
||||
stepsComplete = 0;
|
||||
if (Files.exists(workingDir)) {
|
||||
try (var s = Files.list(workingDir)) {
|
||||
if (s.findAny().isPresent()) throw new IOException("Working directory is not empty!");
|
||||
}
|
||||
}
|
||||
Utils.delete(workingDir); // Delete the directory if it already exists.
|
||||
Files.createDirectory(workingDir);
|
||||
statusListener.messageReceived("Prepared temporary directory for repositories.");
|
||||
List<String> repositoryURIs;
|
||||
try {
|
||||
List<String> repositoryURIs = selector.getURIs();
|
||||
stepsTotal = 2 * repositoryURIs.size();
|
||||
Map<String, Path> repoDirs = downloadRepositories(repositoryURIs);
|
||||
applyActionToRepositories(repoDirs);
|
||||
statusListener.messageReceived("Fetching repository URIs.");
|
||||
repositoryURIs = selector.getURIs();
|
||||
} catch (Exception e) {
|
||||
throw new IOException("Could not fetch repository URIs.", e);
|
||||
}
|
||||
if (repositoryURIs.isEmpty()) {
|
||||
statusListener.messageReceived("No repositories were found.");
|
||||
statusListener.progressUpdated(100f);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
stepsTotal = (finalizationAction == null ? 2 : 3) * repositoryURIs.size();
|
||||
Map<String, Git> repos = downloadRepositories(repositoryURIs);
|
||||
applyActionToRepositories(repos, action);
|
||||
if (finalizationAction != null) {
|
||||
applyActionToRepositories(repos, finalizationAction);
|
||||
}
|
||||
repos.values().stream().filter(Objects::nonNull).forEach(Git::close);
|
||||
} catch (Exception e) {
|
||||
throw new IOException(e);
|
||||
} finally {
|
||||
|
|
@ -60,53 +132,93 @@ public class DistribuGit {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the configured git actions on all selected repositories in an
|
||||
* asynchronous manner.
|
||||
* @return A future that completes when all actions are complete, or if an
|
||||
* error occurs and the operation quits early.
|
||||
*/
|
||||
public CompletableFuture<Void> doActionsAsync() {
|
||||
final CompletableFuture<Void> cf = new CompletableFuture<>();
|
||||
ForkJoinPool.commonPool().submit(() -> {
|
||||
try {
|
||||
doActions();
|
||||
cf.complete(null);
|
||||
} catch (IOException e) {
|
||||
cf.completeExceptionally(e);
|
||||
}
|
||||
});
|
||||
return cf;
|
||||
}
|
||||
|
||||
private void completeStep() {
|
||||
stepsComplete++;
|
||||
statusListener.progressUpdated(stepsComplete / (float) stepsTotal * 100f);
|
||||
}
|
||||
|
||||
private Map<String, Path> downloadRepositories(List<String> uris) throws IOException {
|
||||
Map<String, Path> repositoryDirs = new HashMap<>();
|
||||
/**
|
||||
* Downloads a set of repositories to working directories.
|
||||
* @param uris The repositories to download.
|
||||
* @return A map which maps each repository URI to its {@link Git} instance.
|
||||
* @throws IOException If {@link DistribuGit#strictFail} is set to true,
|
||||
* this will be thrown if an error occurs.
|
||||
*/
|
||||
private Map<String, Git> downloadRepositories(List<String> uris) throws IOException {
|
||||
Map<String, Git> repositoryDirs = new HashMap<>();
|
||||
int dirIdx = 1;
|
||||
for (String repositoryURI : uris) {
|
||||
Path repoDir = workingDir.resolve(Integer.toString(dirIdx++));
|
||||
statusListener.messageReceived("Cloning repository " + repositoryURI + " to " + repoDir);
|
||||
CloneCommand clone = Git.cloneRepository();
|
||||
try {
|
||||
statusListener.messageReceived("Cloning repository " + repositoryURI + " to " + repoDir);
|
||||
CloneCommand clone = Git.cloneRepository();
|
||||
credentials.addCredentials(clone);
|
||||
clone.setDirectory(repoDir.toFile());
|
||||
clone.setURI(repositoryURI);
|
||||
try (var ignored = clone.call()) {
|
||||
repositoryDirs.put(repositoryURI, repoDir);
|
||||
} catch (Exception e) {
|
||||
if (strictFail) {
|
||||
throw new IOException(e);
|
||||
} else {
|
||||
repositoryDirs.put(repositoryURI, null);
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
} catch (Exception e) {
|
||||
if (strictFail) {
|
||||
throw new IOException(e);
|
||||
}
|
||||
statusListener.messageReceived("Could not add credentials to repository: " + e.getMessage());
|
||||
e.printStackTrace();
|
||||
// Skip the rest of the logic since this failed. Just go to the next repository.
|
||||
completeStep();
|
||||
continue;
|
||||
}
|
||||
clone.setDirectory(repoDir.toFile());
|
||||
clone.setURI(repositoryURI);
|
||||
try (var git = clone.call()) {
|
||||
repositoryDirs.put(repositoryURI, git);
|
||||
} catch (Exception e) {
|
||||
if (strictFail) {
|
||||
throw new IOException(e);
|
||||
} else {
|
||||
statusListener.messageReceived("Could not clone repository: " + e.getMessage());
|
||||
repositoryDirs.put(repositoryURI, null);
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
completeStep();
|
||||
}
|
||||
return repositoryDirs;
|
||||
}
|
||||
|
||||
private void applyActionToRepositories(Map<String, Path> repoDirs) throws IOException {
|
||||
for (var entry : repoDirs.entrySet()) {
|
||||
/**
|
||||
* Applies an action to all git repositories in the given map.
|
||||
* @param repositories A map which maps URIs to {@link Git} instances.
|
||||
* @param action The action to apply to each repository.
|
||||
* @throws IOException If {@link DistribuGit#strictFail} is set to true,
|
||||
* this will be thrown if an error occurs.
|
||||
*/
|
||||
private void applyActionToRepositories(Map<String, Git> repositories, RepositoryAction action) throws IOException {
|
||||
for (var entry : repositories.entrySet()) {
|
||||
if (entry.getValue() != null) {
|
||||
try (Git git = Git.open(entry.getValue().toFile())) {
|
||||
try {
|
||||
Git git = entry.getValue();
|
||||
statusListener.messageReceived("Applying action to repository " + entry.getKey());
|
||||
action.doAction(git);
|
||||
} catch (Exception e) {
|
||||
if (strictFail) {
|
||||
throw new IOException(e);
|
||||
}
|
||||
statusListener.messageReceived("Action could not be applied to repository: " + e.getMessage());
|
||||
e.printStackTrace();
|
||||
}
|
||||
} else {
|
||||
|
|
@ -116,9 +228,14 @@ public class DistribuGit {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A builder class to help with constructing {@link DistribuGit} instances
|
||||
* with a fluent method interface.
|
||||
*/
|
||||
public static class Builder {
|
||||
private RepositorySelector selector;
|
||||
private RepositoryAction action;
|
||||
private RepositoryAction finalizationAction;
|
||||
private GitCredentials credentials = cmd -> {};
|
||||
private StatusListener statusListener = new StatusListener() {
|
||||
@Override
|
||||
|
|
@ -145,6 +262,11 @@ public class DistribuGit {
|
|||
return this;
|
||||
}
|
||||
|
||||
public Builder finalizationAction(RepositoryAction finalizationAction) {
|
||||
this.finalizationAction = finalizationAction;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder credentials(GitCredentials credentials) {
|
||||
this.credentials = credentials;
|
||||
return this;
|
||||
|
|
@ -171,7 +293,10 @@ public class DistribuGit {
|
|||
}
|
||||
|
||||
public DistribuGit build() {
|
||||
return new DistribuGit(selector, action, credentials, statusListener, workingDir, strictFail, cleanup);
|
||||
if (selector == null || action == null) {
|
||||
throw new IllegalStateException("Cannot build an instance of DistribuGit without a selector or action.");
|
||||
}
|
||||
return new DistribuGit(selector, action, finalizationAction, credentials, statusListener, workingDir, strictFail, cleanup);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,7 +14,17 @@ import org.eclipse.jgit.util.FS;
|
|||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Path;
|
||||
|
||||
/**
|
||||
* Supplies credentials information to a git command, which is needed when
|
||||
* interacting with private repositories, or when pushing changes to a
|
||||
* repository.
|
||||
*/
|
||||
public interface GitCredentials {
|
||||
/**
|
||||
* Adds credentials information to the given command.
|
||||
* @param gitCommand The command to add credentials to.
|
||||
* @throws Exception If an error occurs while adding credentials.
|
||||
*/
|
||||
void addCredentials(TransportCommand<?, ?> gitCommand) throws Exception;
|
||||
|
||||
static GitCredentials ofUsernamePassword(String username, String password) {
|
||||
|
|
|
|||
|
|
@ -2,13 +2,31 @@ package nl.andrewl.distribugit;
|
|||
|
||||
import org.eclipse.jgit.api.Git;
|
||||
|
||||
/**
|
||||
* An action that can be applied to a git repository.
|
||||
*/
|
||||
public interface RepositoryAction {
|
||||
/**
|
||||
* Performs the action on the given git repository.
|
||||
* @param git A reference to the git repository.
|
||||
* @throws Exception If an error occurs during the action.
|
||||
*/
|
||||
void doAction(Git git) throws Exception;
|
||||
|
||||
/**
|
||||
* An action which executes a system command, as handled by
|
||||
* {@link ProcessBuilder}. Note that the working directory of the command
|
||||
* is set to the directory of the repository.
|
||||
* <p>
|
||||
* Commands invoked via this method have access to the following
|
||||
* environment variables:
|
||||
* </p>
|
||||
* <ul>
|
||||
* <li>DISTRIBUGIT_INVOKE_DIR - The directory in which distribugit
|
||||
* was invoked.</li>
|
||||
* <li>DISTRIBUGIT_WORKING_DIR - The working directory of distribugit,
|
||||
* which contains all repositories.</li>
|
||||
* </ul>
|
||||
* @param command The command to run.
|
||||
* @return The command action.
|
||||
*/
|
||||
|
|
@ -16,6 +34,8 @@ public interface RepositoryAction {
|
|||
return git -> {
|
||||
ProcessBuilder pb = new ProcessBuilder(command);
|
||||
pb.directory(git.getRepository().getWorkTree());
|
||||
pb.environment().put("DISTRIBUGIT_INVOKE_DIR", pb.directory().getParentFile().getParentFile().getAbsolutePath());
|
||||
pb.environment().put("DISTRIBUGIT_WORKING_DIR", pb.directory().getParentFile().getAbsolutePath());
|
||||
pb.inheritIO();
|
||||
Process p = pb.start();
|
||||
int result = p.waitFor();
|
||||
|
|
|
|||
|
|
@ -9,6 +9,11 @@ import java.util.List;
|
|||
* Component which produces a list of repositories to operate on.
|
||||
*/
|
||||
public interface RepositorySelector {
|
||||
/**
|
||||
* Gets a list of repository URIs to operate on.
|
||||
* @return A list of repository URIs.
|
||||
* @throws Exception If an error occurs while fetching the URIs.
|
||||
*/
|
||||
List<String> getURIs() throws Exception;
|
||||
|
||||
static RepositorySelector fromCollection(Collection<String> uris) {
|
||||
|
|
@ -18,4 +23,12 @@ public interface RepositorySelector {
|
|||
static RepositorySelector from(String... uris) {
|
||||
return () -> List.of(uris);
|
||||
}
|
||||
|
||||
static RepositorySelector from(RepositorySelector... selectors) {
|
||||
return () -> {
|
||||
List<String> uris = new ArrayList<>();
|
||||
for (var selector : selectors) uris.addAll(selector.getURIs());
|
||||
return uris;
|
||||
};
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,18 @@
|
|||
package nl.andrewl.distribugit;
|
||||
|
||||
/**
|
||||
* Listens for updates during {@link DistribuGit#doActions()}.
|
||||
*/
|
||||
public interface StatusListener {
|
||||
/**
|
||||
* Called when the operation's progress is updated.
|
||||
* @param percentage The percentage (0 - 100) complete.
|
||||
*/
|
||||
void progressUpdated(float percentage);
|
||||
|
||||
/**
|
||||
* Called when the DistribuGit operation emits a message.
|
||||
* @param message The message that was emitted.
|
||||
*/
|
||||
void messageReceived(String message);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,89 @@
|
|||
package nl.andrewl.distribugit.cli;
|
||||
|
||||
import nl.andrewl.distribugit.DistribuGit;
|
||||
import nl.andrewl.distribugit.GitCredentials;
|
||||
import nl.andrewl.distribugit.RepositoryAction;
|
||||
import nl.andrewl.distribugit.StatusListener;
|
||||
import picocli.CommandLine;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.concurrent.Callable;
|
||||
|
||||
@CommandLine.Command(name = "distribugit", description = "DistribuGit command-line tool for performing distributed git operations.", mixinStandardHelpOptions = true)
|
||||
public class DistribuGitCommand implements Callable<Integer> {
|
||||
@CommandLine.Option(names = {"-d", "--dir"}, description = "The working directory for DistribuGit", defaultValue = "./.distribugit_tmp")
|
||||
public Path workingDir;
|
||||
|
||||
@CommandLine.Option(
|
||||
names = {"-s", "--selector"},
|
||||
description = """
|
||||
The repository selector to use.
|
||||
The following selector types are permitted:
|
||||
- "org-repo-prefix:{orgName}/{repoPrefix}": Selects repositories from a GitHub organization whose name begins with the given prefix.
|
||||
- "stdin": Selects repository URIs that have been provided to the program via stdin, with one URI per line.
|
||||
- "file:{filePath}": Selects repository URIs written in a file, with one URI per line.
|
||||
""",
|
||||
required = true
|
||||
)
|
||||
public String selectorExpression;
|
||||
|
||||
@CommandLine.Option(
|
||||
names = {"-a", "--action"},
|
||||
description = """
|
||||
The command to run on each repository.
|
||||
Each command has access to the following environment variables:
|
||||
- DISTRIBUGIT_INVOKE_DIR - The directory in which distribugit was invoked.
|
||||
- DISTRIBUGIT_WORKING_DIR - The working directory of distribugit.
|
||||
""",
|
||||
required = true
|
||||
)
|
||||
public String actionCommand;
|
||||
|
||||
@CommandLine.Option(
|
||||
names = {"-fa", "--finalization-action"},
|
||||
description = "A command to run on each repository after all normal actions. Has access to the same environment variables as --action."
|
||||
)
|
||||
public String finalizationActionCommand;
|
||||
|
||||
@CommandLine.Option(names = {"-t", "--access-token"}, description = "The access token to use to perform operations.")
|
||||
public String accessToken;
|
||||
|
||||
@CommandLine.Option(names = {"-sf", "--strict-fail"}, description = "Whether to preemptively fail if any error occurs.", defaultValue = "true")
|
||||
public boolean strictFail;
|
||||
|
||||
@CommandLine.Option(names = {"-cl", "--cleanup"}, description = "Whether to remove all repository files when done.", defaultValue = "false")
|
||||
public boolean cleanup;
|
||||
|
||||
@Override
|
||||
public Integer call() throws Exception {
|
||||
var builder = new DistribuGit.Builder()
|
||||
.workingDir(workingDir)
|
||||
.strictFail(strictFail)
|
||||
.cleanup(cleanup)
|
||||
.selector(SelectorExpressionParser.parse(selectorExpression, accessToken))
|
||||
.action(RepositoryAction.ofCommand(actionCommand.split("\\s+")));
|
||||
if (finalizationActionCommand != null) {
|
||||
builder.finalizationAction(RepositoryAction.ofCommand(finalizationActionCommand.split("\\s+")));
|
||||
}
|
||||
if (accessToken != null) {
|
||||
builder.credentials(GitCredentials.ofUsernamePassword(accessToken, ""));
|
||||
}
|
||||
builder.statusListener(new StatusListener() {
|
||||
@Override
|
||||
public void progressUpdated(float percentage) {
|
||||
System.out.printf("Progress: %.1f%%%n", percentage);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void messageReceived(String message) {
|
||||
System.out.println(message);
|
||||
}
|
||||
});
|
||||
builder.build().doActions();
|
||||
return 0;
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
new CommandLine(new DistribuGitCommand()).execute(args);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
package nl.andrewl.distribugit.cli;
|
||||
|
||||
import nl.andrewl.distribugit.RepositorySelector;
|
||||
import nl.andrewl.distribugit.selectors.GitHubSelectorBuilder;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStreamReader;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
public class SelectorExpressionParser {
|
||||
private static final Pattern SELECTOR_EXPRESSION_PATTERN = Pattern.compile("([\\w-]+):(.+)");
|
||||
private static final Pattern ORG_REPO_PREFIX_PATTERN = Pattern.compile("(.+)/(.+)");
|
||||
|
||||
public static RepositorySelector parse(String expr, String accessToken) throws IOException {
|
||||
Matcher m = SELECTOR_EXPRESSION_PATTERN.matcher(expr);
|
||||
if (!m.find()) throw new IllegalArgumentException("Invalid selector expression. Should be \"selector-type[:expression]\".");
|
||||
String slug = m.group(1);
|
||||
String content = m.groupCount() > 1 ? m.group(2) : null;
|
||||
return switch (slug) {
|
||||
case "org-repo-prefix" -> parseOrgRepoPrefix(content, accessToken);
|
||||
case "stdin" -> stdinSelector();
|
||||
case "file" -> fileSelector(content);
|
||||
default -> throw new IllegalArgumentException("Unsupported selector type: " + slug);
|
||||
};
|
||||
}
|
||||
|
||||
private static RepositorySelector parseOrgRepoPrefix(String content, String accessToken) throws IOException {
|
||||
if (content == null) throw new IllegalArgumentException("Missing required selector expression.");
|
||||
if (accessToken == null) throw new IllegalArgumentException("Missing required access-token for GitHub org-repo-prefix selector.");
|
||||
Matcher m = ORG_REPO_PREFIX_PATTERN.matcher(content);
|
||||
if (!m.find()) throw new IllegalArgumentException("Invalid content for org-repo-prefix select. Should be \"orgName/prefix\"");
|
||||
return GitHubSelectorBuilder.fromPersonalAccessToken(accessToken).orgAndPrefix(m.group(1), m.group(2));
|
||||
}
|
||||
|
||||
private static RepositorySelector stdinSelector() {
|
||||
return () -> {
|
||||
List<String> uris = new ArrayList<>();
|
||||
String line;
|
||||
try (var reader = new BufferedReader(new InputStreamReader(System.in))) {
|
||||
while ((line = reader.readLine()) != null) {
|
||||
if (!line.isBlank()) {
|
||||
uris.add(line.trim());
|
||||
}
|
||||
}
|
||||
}
|
||||
return uris;
|
||||
};
|
||||
}
|
||||
|
||||
private static RepositorySelector fileSelector(String content) {
|
||||
if (content == null) throw new IllegalArgumentException("No file paths were given.");
|
||||
String[] filePaths = content.split(";");
|
||||
if (filePaths.length < 1) throw new IllegalArgumentException("No file paths were given.");
|
||||
List<Path> paths = Arrays.stream(filePaths).map(Path::of).toList();
|
||||
for (var path : paths) {
|
||||
if (Files.notExists(path)) throw new IllegalArgumentException("File " + path + " does not exist.");
|
||||
if (!Files.isRegularFile(path)) throw new IllegalArgumentException("File " + path + " is not a regular file.");
|
||||
if (!Files.isReadable(path)) throw new IllegalArgumentException("File " + path + " is not readable.");
|
||||
}
|
||||
return () -> {
|
||||
List<String> uris = new ArrayList<>();
|
||||
for (var path : paths) {
|
||||
try (var s = Files.lines(path)) {
|
||||
uris.addAll(s.filter(str -> !str.isBlank()).toList());
|
||||
}
|
||||
}
|
||||
return uris;
|
||||
};
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
package nl.andrewl.distribugit.selectors;
|
||||
|
||||
import org.kohsuke.github.GHRepository;
|
||||
import org.kohsuke.github.GitHub;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public interface GitHubRepositorySelector {
|
||||
List<GHRepository> getRepos(GitHub gh) throws Exception;
|
||||
}
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
package nl.andrewl.distribugit.selectors;
|
||||
|
||||
import nl.andrewl.distribugit.RepositorySelector;
|
||||
import org.kohsuke.github.GHOrganization;
|
||||
import org.kohsuke.github.GHRepository;
|
||||
import org.kohsuke.github.GitHub;
|
||||
import org.kohsuke.github.GitHubBuilder;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* A builder that can be used to construct {@link RepositorySelector} instances
|
||||
* that fetch repositories from GitHub via its API.
|
||||
*/
|
||||
public record GitHubSelectorBuilder(GitHub gh) {
|
||||
public static GitHubSelectorBuilder fromPersonalAccessToken(String token) throws IOException {
|
||||
return fromPersonalAccessToken(token, null);
|
||||
}
|
||||
|
||||
public static GitHubSelectorBuilder fromPersonalAccessToken(String token, String userOrOrgName) throws IOException {
|
||||
return new GitHubSelectorBuilder(new GitHubBuilder().withOAuthToken(token, userOrOrgName).build());
|
||||
}
|
||||
|
||||
/**
|
||||
* Select repositories using an organization name, and a prefix to match
|
||||
* against all repositories in that organization.
|
||||
* @param orgName The name of the organization.
|
||||
* @param prefix The prefix to use.
|
||||
* @return A selector that selects matching repositories.
|
||||
*/
|
||||
public RepositorySelector orgAndPrefix(String orgName, String prefix) {
|
||||
return () -> {
|
||||
List<String> repoURIs = new ArrayList<>();
|
||||
GHOrganization org = gh.getOrganization(orgName);
|
||||
for (GHRepository repo : org.listRepositories()) {
|
||||
if (repo.getName().startsWith(prefix)) {
|
||||
repoURIs.add(repo.getHttpTransportUrl());
|
||||
}
|
||||
}
|
||||
return repoURIs;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A custom selector that can be used to perform some operations using the
|
||||
* GitHub API and return a list of repositories.
|
||||
* @param selector The GitHub selector to use.
|
||||
* @return A selector that selects GitHub repositories.
|
||||
*/
|
||||
public RepositorySelector custom(GitHubRepositorySelector selector) {
|
||||
return () -> selector.getRepos(gh).stream().map(GHRepository::getHttpTransportUrl).toList();
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
# SLF4J's SimpleLogger configuration file
|
||||
# Simple implementation of Logger that sends all enabled log messages, for all defined loggers, to System.err.
|
||||
|
||||
# Default logging detail level for all instances of SimpleLogger.
|
||||
# Must be one of ("trace", "debug", "info", "warn", or "error").
|
||||
# If not specified, defaults to "info".
|
||||
org.slf4j.simpleLogger.defaultLogLevel=error
|
||||
|
||||
# Logging detail level for a SimpleLogger instance named "xxxxx".
|
||||
# Must be one of ("trace", "debug", "info", "warn", or "error").
|
||||
# If not specified, the default logging detail level is used.
|
||||
#org.slf4j.simpleLogger.log.xxxxx=
|
||||
|
||||
# Set to true if you want the current date and time to be included in output messages.
|
||||
# Default is false, and will output the number of milliseconds elapsed since startup.
|
||||
#org.slf4j.simpleLogger.showDateTime=false
|
||||
|
||||
# The date and time format to be used in the output messages.
|
||||
# The pattern describing the date and time format is the same that is used in java.text.SimpleDateFormat.
|
||||
# If the format is not specified or is invalid, the default format is used.
|
||||
# The default format is yyyy-MM-dd HH:mm:ss:SSS Z.
|
||||
#org.slf4j.simpleLogger.dateTimeFormat=yyyy-MM-dd HH:mm:ss:SSS Z
|
||||
|
||||
# Set to true if you want to output the current thread name.
|
||||
# Defaults to true.
|
||||
#org.slf4j.simpleLogger.showThreadName=true
|
||||
|
||||
# Set to true if you want the Logger instance name to be included in output messages.
|
||||
# Defaults to true.
|
||||
#org.slf4j.simpleLogger.showLogName=true
|
||||
|
||||
# Set to true if you want the last component of the name to be included in output messages.
|
||||
# Defaults to false.
|
||||
#org.slf4j.simpleLogger.showShortLogName=false
|
||||
|
|
@ -0,0 +1,79 @@
|
|||
package nl.andrewl.distribugit;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Path;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
|
||||
public class DistribuGitTest {
|
||||
@Test
|
||||
public void testBasicActionOperation() throws IOException {
|
||||
AtomicInteger count = new AtomicInteger(0);
|
||||
RepositoryAction countAction = git -> count.incrementAndGet();
|
||||
new DistribuGit.Builder()
|
||||
.selector(RepositorySelector.from(
|
||||
"https://github.com/andrewlalis/record-net.git",
|
||||
"https://github.com/andrewlalis/distribugit.git",
|
||||
"https://github.com/andrewlalis/RandomHotbar.git"
|
||||
))
|
||||
.action(countAction)
|
||||
.statusListener(noOpListener())
|
||||
.strictFail(true)
|
||||
.cleanup(true)
|
||||
.workingDir(Path.of(".", ".distribugit_test_tmp"))
|
||||
.build().doActions();
|
||||
assertEquals(3, count.get());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testStrictFailFalse() throws IOException {
|
||||
AtomicInteger count = new AtomicInteger(0);
|
||||
RepositoryAction countAction = git -> count.incrementAndGet();
|
||||
new DistribuGit.Builder()
|
||||
.selector(RepositorySelector.from(
|
||||
"https://github.com/andrewlalis/record-net.git",
|
||||
"https://github.com/andrewlalis/distribugit.git",
|
||||
"https://github.com/andrewlalis/somerandomgitrepositorythatdoesntexist.git"
|
||||
))
|
||||
.action(countAction)
|
||||
.statusListener(noOpListener())
|
||||
.strictFail(false)
|
||||
.cleanup(true)
|
||||
.workingDir(Path.of(".", ".distribugit_test_tmp"))
|
||||
.build().doActions();
|
||||
assertEquals(2, count.get());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testStrictFailTrue() {
|
||||
AtomicInteger count = new AtomicInteger(0);
|
||||
RepositoryAction countAction = git -> count.incrementAndGet();
|
||||
DistribuGit d = new DistribuGit.Builder()
|
||||
.selector(RepositorySelector.from(
|
||||
"https://github.com/andrewlalis/record-net.git",
|
||||
"https://github.com/andrewlalis/distribugit.git",
|
||||
"https://github.com/andrewlalis/somerandomgitrepositorythatdoesntexist.git"
|
||||
))
|
||||
.action(countAction)
|
||||
.statusListener(noOpListener())
|
||||
.strictFail(true)
|
||||
.cleanup(true)
|
||||
.workingDir(Path.of(".", ".distribugit_test_tmp"))
|
||||
.build();
|
||||
assertThrows(IOException.class, d::doActions);
|
||||
}
|
||||
|
||||
private static StatusListener noOpListener() {
|
||||
return new StatusListener() {
|
||||
@Override
|
||||
public void progressUpdated(float percentage) {}
|
||||
|
||||
@Override
|
||||
public void messageReceived(String message) {}
|
||||
};
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue