diff --git a/docs/de/user/repo/assets/import-repository.png b/docs/de/user/repo/assets/import-repository.png index 90b02347ea..d7f78837aa 100644 Binary files a/docs/de/user/repo/assets/import-repository.png and b/docs/de/user/repo/assets/import-repository.png differ diff --git a/docs/de/user/repo/index.md b/docs/de/user/repo/index.md index c62cde235d..a12aee790a 100644 --- a/docs/de/user/repo/index.md +++ b/docs/de/user/repo/index.md @@ -37,17 +37,29 @@ neu erstellt werden. ![Repository erstellen](assets/create-repository.png) ### Repository importieren -Neben dem Erstellen von neuen Repository können auch bestehende Repository in SCM-Manager importiert werden. +Neben dem Erstellen von neuen Repository können auch bestehende Repository in den SCM-Manager importiert werden. Wechseln Sie über den Schalter oben rechts auf die Importseite und füllen Sie die benötigten Informationen aus. -Das gewählte Repository wird zum SCM-Manager hinzugefügt und sämtliche Repository Daten inklusive aller Branches und Tags werden importiert. -Zusätzlich zum normalen Repository Import gibt es die Möglichkeit ein Repository Archiv mit Metadaten zu importieren. -Dieses Repository Archiv muss von einem anderen SCM-Manager exportiert worden sein und wird vor dem Import auf -Kompatibilität der Daten überprüft (der SCM-Manager und alle installierten Plugins müssen mindestens die Version des -exportierenden Systems haben). -Ist die zu importierende Datei verschlüsselt, muss das korrekte Passwort zum Entschlüsseln mitgeliefert werden. +In Abhängigkeit vom Typen des zu importierenden Repositories gibt es verschiedene Möglichkeiten: +- **Import via URL** (nur Git und Mercurial): Hier wird ein Repository von einem anderen Server über die gegebene URL + importiert. Zusätzlich kann ein Benutzername und ein Passwort für die Authentifizierung angegeben werden. Für Git + Repositories kann darüber hinaus der Import von LFS Dateien (falls vorhanden) ausgeschlossen werden. +- **Import aus Dump ohne Metadaten**: Hier kann eine Datei hochgeladen werden. Dieses kann entweder ein einfacher Export + aus einer anderen SCM-Manager Instanz sein oder ein Dump aus einem anderen Repository: + - Für Git und Mercurial muss es das per Tar gepackte "interne" Verzeichnis sein (das `.git` bzw. das `.hg` Verzeichnis). + - Für SVN kann ein per `svnamdin` erstellter Dump genutzt werden. + Wenn diese Dateien per Zip komprimiert sind, muss die Option "Komprimiert" gewählt werden. +- **Import aus SCM-Manager-Dump mit Metadaten**: Mit dieser Option können Exporte mit Metadaten aus anderen SCM-Manager + Instanzen importiert werden. Dieses Repository Archiv wird vor dem Import auf + Kompatibilität der Daten überprüft (der SCM-Manager und alle installierten Plugins müssen mindestens die Version des + exportierenden Systems haben). + +Ist die zu importierende Datei verschlüsselt, muss das korrekte Passwort zum Entschlüsseln mitgeliefert werden. Wird kein Passwort gesetzt, geht der SCM-Manager davon aus, dass die Datei unverschlüsselt ist. +Das gewählte Repository wird zum SCM-Manager hinzugefügt und sämtliche Repository Daten inklusive aller Branches und Tags werden importiert. + + ![Repository importieren](assets/import-repository.png) ### Repository Informationen diff --git a/docs/en/user/repo/assets/import-repository.png b/docs/en/user/repo/assets/import-repository.png index f2967400db..4ae5d2f73b 100644 Binary files a/docs/en/user/repo/assets/import-repository.png and b/docs/en/user/repo/assets/import-repository.png differ diff --git a/docs/en/user/repo/index.md b/docs/en/user/repo/index.md index 7f4411b18d..0388f0e0ce 100644 --- a/docs/en/user/repo/index.md +++ b/docs/en/user/repo/index.md @@ -33,17 +33,29 @@ be chosen. To create a new namespace, this has to be chosen from the drop down e ![Create Repository](assets/create-repository.png) ### Import a Repository -Beneath creating new repositories you also may import existing repositories to SCM-Manager. +Besided creating new repositories you also may import existing repositories to SCM-Manager. Just use the Switcher on top right to navigate to the import page and fill the import wizard with the required information. -Your repository will be added to SCM-Manager and all repository data including all branches and tags will be imported. -In addition to the normal repository import, there is the possibility to import a repository archive with metadata. -This repository archive must have been exported from another SCM-Manager and is checked for data compatibility before -import (the SCM-Manager and all its installed plugins have to have at least the versions of the system the export has -been created on). -If the file to be imported is encrypted, the correct password must be supplied for decryption. +Depending on the type of the new repository, there are different "ways" to import repositories: +- **Import via URL** (Git and Mercurial only): Here the repository is imported from a remote server specified by the + remote repository's URL. In addition, a username and a password can be provided. Furthermore, for Git repositories + LFS files can be excluded (if LFS is used in the remote repository). +- **Import from dump without metadata**: Here a file can be uploaded. This may be a simple export from another SCN-Manager + instance, or a dump file from another repository: + - For Git and Mercurial, this has to be a tar packed file of the "internal" repository (the `.git` or the + `.hg` directory). + - For SVN a standard dump file created with `svnadmin` can be used. + If the import file is zip compressed, check the "compressed" option. +- **Import from SCM-Manager dump with metadata**: This option is to be used with exports of repositories with metadata + from other SCM-Manager instances. It will be checked for data compatibility before + import (the SCM-Manager and all its installed plugins have to have at least the versions of the system the export has + been created on). + +File dumps can be encrypted if they were exported from an SCM-Manager. In this case, the password has to be specified. If no password is set, the SCM Manager assumes that the file is unencrypted. +Your repository will be added to SCM-Manager and all repository data including all branches and tags will be imported. + ![Import Repository](assets/import-repository.png) ### Repository Information diff --git a/gradle/changelog/lfs_import.yaml b/gradle/changelog/lfs_import.yaml new file mode 100644 index 0000000000..a25b7fb02d --- /dev/null +++ b/gradle/changelog/lfs_import.yaml @@ -0,0 +1,2 @@ +- type: added + description: Git import with lfs support ([#2133](https://github.com/scm-manager/scm-manager/pull/2133)) diff --git a/scm-core/src/main/java/sonia/scm/notifications/Notification.java b/scm-core/src/main/java/sonia/scm/notifications/Notification.java index 26bded357e..ec4e85c7ca 100644 --- a/scm-core/src/main/java/sonia/scm/notifications/Notification.java +++ b/scm-core/src/main/java/sonia/scm/notifications/Notification.java @@ -27,6 +27,7 @@ package sonia.scm.notifications; import lombok.Value; import java.time.Instant; +import java.util.Map; /** * Notifications can be used to send a message to specific user. @@ -39,16 +40,22 @@ public class Notification { Type type; String link; String message; + Map parameters; Instant createdAt; public Notification(Type type, String link, String message) { - this(type, link, message, Instant.now()); + this(type, link, message, null); } - Notification(Type type, String link, String message, Instant createdAt) { + public Notification(Type type, String link, String message, Map parameters) { + this(type, link, message, parameters, Instant.now()); + } + + Notification(Type type, String link, String message, Map parameters, Instant createdAt) { this.type = type; this.link = link; this.message = message; + this.parameters = parameters; this.createdAt = createdAt; } diff --git a/scm-core/src/main/java/sonia/scm/repository/api/PullCommandBuilder.java b/scm-core/src/main/java/sonia/scm/repository/api/PullCommandBuilder.java index a86b66c899..80146d085a 100644 --- a/scm-core/src/main/java/sonia/scm/repository/api/PullCommandBuilder.java +++ b/scm-core/src/main/java/sonia/scm/repository/api/PullCommandBuilder.java @@ -86,6 +86,19 @@ public final class PullCommandBuilder { return this; } + /** + * Set whether to fetch LFS files (true) or not (false). + * This may not work for all repository types. + * + * @param fetchLfs Whether to fetch LFS files or not + * @return this builder instance. + * @since 2.40.0 + */ + public PullCommandBuilder doFetchLfs(boolean fetchLfs) { + request.setFetchLfs(fetchLfs); + return this; + } + /** * Pull all changes from the given remote url. * diff --git a/scm-core/src/main/java/sonia/scm/repository/api/PullResponse.java b/scm-core/src/main/java/sonia/scm/repository/api/PullResponse.java index adf6ce1dc2..b49826334f 100644 --- a/scm-core/src/main/java/sonia/scm/repository/api/PullResponse.java +++ b/scm-core/src/main/java/sonia/scm/repository/api/PullResponse.java @@ -21,11 +21,13 @@ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE * SOFTWARE. */ - + package sonia.scm.repository.api; +import lombok.Value; + /** - * The {@link PullResponse} is the result of the + * The {@link PullResponse} is the result of the * {@link PullCommandBuilder#pull(sonia.scm.repository.Repository)} method and * contains informations over the executed pull command. * @@ -35,20 +37,53 @@ package sonia.scm.repository.api; public final class PullResponse extends AbstractPushOrPullResponse { - /** - * Constructs a new PullResponse. - * - */ - public PullResponse() {} + private final LfsCount lfsCount; /** * Constructs a new PullResponse. * + */ + public PullResponse() { + this.lfsCount = new LfsCount(0, 0); + } + + /** + * Constructs a new PullResponse. * * @param changesetCount count of pulled changesets */ public PullResponse(long changesetCount) { + this(changesetCount, new LfsCount(0, 0)); + } + + /** + * Constructs a new PullResponse. + * + * @param changesetCount count of pulled changesets + * @param lfsCount Object for the count of potentially loaded lfs files + */ + public PullResponse(long changesetCount, LfsCount lfsCount) { super(changesetCount); + this.lfsCount = lfsCount; + } + + /** + * Object for the count of potentially loaded lfs files. + */ + public LfsCount getLfsCount() { + return lfsCount; + } + + @Value + public static class LfsCount { + /** + * Count of successfully loaded lfs files. + */ + int successCount; + /** + * Count of failed lfs files. + */ + int failureCount; } } diff --git a/scm-core/src/main/java/sonia/scm/repository/spi/PullCommandRequest.java b/scm-core/src/main/java/sonia/scm/repository/spi/PullCommandRequest.java index f42eb0261e..3697a0eab0 100644 --- a/scm-core/src/main/java/sonia/scm/repository/spi/PullCommandRequest.java +++ b/scm-core/src/main/java/sonia/scm/repository/spi/PullCommandRequest.java @@ -21,13 +21,27 @@ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE * SOFTWARE. */ - + package sonia.scm.repository.spi; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.Setter; +import lombok.ToString; + /** * Request object for {@link PullCommand}. - * + * * @author Sebastian Sdorra * @since 1.31 */ -public final class PullCommandRequest extends RemoteCommandRequest {} +@Getter +@Setter +@EqualsAndHashCode(callSuper = true) +@ToString +public final class PullCommandRequest extends RemoteCommandRequest { + /** + * @since 2.40.0 + */ + private boolean fetchLfs; +} diff --git a/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/GitMirrorCommand.java b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/GitMirrorCommand.java index 8acaf36782..7aff4af9a1 100644 --- a/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/GitMirrorCommand.java +++ b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/GitMirrorCommand.java @@ -60,8 +60,8 @@ import sonia.scm.repository.api.MirrorCommandResult.ResultType; import sonia.scm.repository.api.MirrorFilter; import sonia.scm.repository.api.MirrorFilter.Result; import sonia.scm.repository.api.UsernamePasswordCredential; +import sonia.scm.repository.spi.LfsLoader.LfsLoaderLogger; import sonia.scm.store.ConfigurationStore; -import sonia.scm.web.lfs.LfsBlobStoreFactory; import javax.inject.Inject; import java.io.IOException; @@ -367,7 +367,16 @@ public class GitMirrorCommand extends AbstractGitCommand implements MirrorComman logger.logChange(ref, referenceName, getUpdateType(ref)); if (!mirrorCommandRequest.isIgnoreLfs()) { - lfsLoader.inspectTree(ref.getNewObjectId(), mirrorCommandRequest, git.getRepository(), mirrorLog, lfsUpdateResult, repository); + LfsLoaderLogger lfsLoaderLogger = new MirrorLfsLoaderLogger(); + lfsLoader.inspectTree( + ref.getNewObjectId(), + git.getRepository(), + lfsLoaderLogger, + lfsUpdateResult, + repository, + mirrorHttpConnectionProvider.createHttpConnectionFactory(mirrorCommandRequest, mirrorLog), + mirrorCommandRequest.getSourceUrl() + ); } } } @@ -406,6 +415,19 @@ public class GitMirrorCommand extends AbstractGitCommand implements MirrorComman private String getUpdateType(TrackingRefUpdate trackingRefUpdate) { return trackingRefUpdate.getResult().name().toLowerCase(Locale.ENGLISH); } + + private class MirrorLfsLoaderLogger implements LfsLoaderLogger { + @Override + public void failed(Exception e) { + mirrorLog.add("Failed to load lfs file:"); + mirrorLog.add(e.getMessage()); + } + + @Override + public void loading(String name) { + mirrorLog.add(String.format("Loading lfs file with id '%s'", name)); + } + } } private class LoggerWithHeader { @@ -653,7 +675,7 @@ public class GitMirrorCommand extends AbstractGitCommand implements MirrorComman } private boolean isOfTypeOrEmpty(Optional updateType, MirrorFilter.UpdateType type) { - return !updateType.isPresent() || updateType.get() == type; + return updateType.isEmpty() || updateType.get() == type; } private Optional getUpdateTypeFor(ReceiveCommand receiveCommand) { diff --git a/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/GitPullCommand.java b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/GitPullCommand.java index 317a2a8a5e..e702796c7e 100644 --- a/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/GitPullCommand.java +++ b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/GitPullCommand.java @@ -41,6 +41,7 @@ import sonia.scm.repository.GitRepositoryHandler; import sonia.scm.repository.GitUtil; import sonia.scm.repository.Repository; import sonia.scm.repository.api.ImportFailedException; +import sonia.scm.repository.api.MirrorCommandResult; import sonia.scm.repository.api.PullResponse; import javax.inject.Inject; @@ -54,14 +55,21 @@ public class GitPullCommand extends AbstractGitPushOrPullCommand implements PullCommand { private static final Logger LOG = LoggerFactory.getLogger(GitPullCommand.class); + private final PostReceiveRepositoryHookEventFactory postReceiveRepositoryHookEventFactory; + private final LfsLoader lfsLoader; + private final PullHttpConnectionProvider pullHttpConnectionProvider; @Inject public GitPullCommand(GitRepositoryHandler handler, GitContext context, - PostReceiveRepositoryHookEventFactory postReceiveRepositoryHookEventFactory) { + PostReceiveRepositoryHookEventFactory postReceiveRepositoryHookEventFactory, + LfsLoader lfsLoader, + PullHttpConnectionProvider pullHttpConnectionProvider) { super(handler, context); this.postReceiveRepositoryHookEventFactory = postReceiveRepositoryHookEventFactory; + this.lfsLoader = lfsLoader; + this.pullHttpConnectionProvider = pullHttpConnectionProvider; } @Override @@ -81,7 +89,7 @@ public class GitPullCommand extends AbstractGitPushOrPullCommand return response; } - private PullResponse convert(Git git, FetchResult fetch) { + private PullResponse convert(Git git, FetchResult fetch, CountingLfsLoaderLogger lfsLoaderLogger) { long counter = 0; for (TrackingRefUpdate tru : fetch.getTrackingRefUpdates()) { @@ -90,7 +98,7 @@ public class GitPullCommand extends AbstractGitPushOrPullCommand LOG.debug("received {} changesets by pull", counter); - return new PullResponse(counter); + return new PullResponse(counter, new PullResponse.LfsCount(lfsLoaderLogger.getSuccessCount(), lfsLoaderLogger.getFailureCount())); } private long count(Git git, TrackingRefUpdate tru) { @@ -178,7 +186,11 @@ public class GitPullCommand extends AbstractGitPushOrPullCommand .call(); //J+ - response = convert(git, result); + CountingLfsLoaderLogger lfsLoaderLogger = new CountingLfsLoaderLogger(); + if (request.isFetchLfs()) { + fetchLfs(request, git, lfsLoaderLogger); + } + response = convert(git, result, lfsLoaderLogger); } catch (GitAPIException ex) { throw new ImportFailedException( @@ -193,7 +205,45 @@ public class GitPullCommand extends AbstractGitPushOrPullCommand return response; } + private void fetchLfs(PullCommandRequest request, Git git, LfsLoader.LfsLoaderLogger lfsLoaderLogger) throws IOException { + open().getRefDatabase().getRefs().forEach( + ref -> lfsLoader.inspectTree( + ref.getObjectId(), + git.getRepository(), + lfsLoaderLogger, + new MirrorCommandResult.LfsUpdateResult(), + repository, + pullHttpConnectionProvider.createHttpConnectionFactory(request), + request.getRemoteUrl().toString() + ) + ); + } + private void firePostReceiveRepositoryHookEvent(Git git, FetchResult result) { postReceiveRepositoryHookEventFactory.fireForFetch(git, result); } + + private static class CountingLfsLoaderLogger implements LfsLoader.LfsLoaderLogger { + + private int successCount = 0; + private int failureCount = 0; + + @Override + public void failed(Exception e) { + ++failureCount; + } + + @Override + public void loading(String name) { + ++successCount; + } + + public int getSuccessCount() { + return successCount; + } + + public int getFailureCount() { + return failureCount; + } + } } diff --git a/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/LfsLoader.java b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/LfsLoader.java index 85f0301d9b..d023c0cf27 100644 --- a/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/LfsLoader.java +++ b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/LfsLoader.java @@ -50,34 +50,39 @@ import java.io.InputStream; import java.nio.file.Files; import java.nio.file.Path; import java.util.Collection; -import java.util.List; class LfsLoader { private static final Logger LOG = LoggerFactory.getLogger(LfsLoader.class); private final LfsBlobStoreFactory lfsBlobStoreFactory; - private final MirrorHttpConnectionProvider mirrorHttpConnectionProvider; @Inject - LfsLoader(LfsBlobStoreFactory lfsBlobStoreFactory, MirrorHttpConnectionProvider mirrorHttpConnectionProvider) { + LfsLoader(LfsBlobStoreFactory lfsBlobStoreFactory) { this.lfsBlobStoreFactory = lfsBlobStoreFactory; - this.mirrorHttpConnectionProvider = mirrorHttpConnectionProvider; } void inspectTree(ObjectId newObjectId, - MirrorCommandRequest mirrorCommandRequest, Repository gitRepository, - List mirrorLog, + LfsLoaderLogger mirrorLog, LfsUpdateResult lfsUpdateResult, - sonia.scm.repository.Repository repository) { - String mirrorUrl = mirrorCommandRequest.getSourceUrl(); - EntryHandler entryHandler = new EntryHandler(repository, gitRepository, mirrorCommandRequest, mirrorLog, lfsUpdateResult); + sonia.scm.repository.Repository repository, + HttpConnectionFactory httpConnectionFactory, + String url) { + EntryHandler entryHandler = new EntryHandler(repository, gitRepository, mirrorLog, lfsUpdateResult, httpConnectionFactory); + inspectTree(newObjectId, entryHandler, gitRepository, mirrorLog, lfsUpdateResult, url); + } + private void inspectTree(ObjectId newObjectId, + EntryHandler entryHandler, + Repository gitRepository, + LfsLoaderLogger mirrorLog, + LfsUpdateResult lfsUpdateResult, + String sourceUrl) { try { gitRepository .getConfig() - .setString(ConfigConstants.CONFIG_SECTION_LFS, null, ConfigConstants.CONFIG_KEY_URL, computeLfsUrl(mirrorUrl)); + .setString(ConfigConstants.CONFIG_SECTION_LFS, null, ConfigConstants.CONFIG_KEY_URL, computeLfsUrl(sourceUrl)); TreeWalk treeWalk = new TreeWalk(gitRepository); treeWalk.setFilter(new LfsPointerFilter()); @@ -94,17 +99,16 @@ class LfsLoader { } } catch (Exception e) { LOG.warn("failed to load lfs files", e); - mirrorLog.add("Failed to load lfs files:"); - mirrorLog.add(e.getMessage()); + mirrorLog.failed(e); lfsUpdateResult.increaseFailureCount(); } } - private String computeLfsUrl(String mirrorUrl) { - if (mirrorUrl.endsWith(".git")) { - return mirrorUrl + Protocol.INFO_LFS_ENDPOINT; + private String computeLfsUrl(String sourceUrl) { + if (sourceUrl.endsWith(".git")) { + return sourceUrl + Protocol.INFO_LFS_ENDPOINT; } else { - return mirrorUrl + ".git" + Protocol.INFO_LFS_ENDPOINT; + return sourceUrl + ".git" + Protocol.INFO_LFS_ENDPOINT; } } @@ -112,22 +116,22 @@ class LfsLoader { private final BlobStore lfsBlobStore; private final Repository gitRepository; - private final List mirrorLog; + private final LfsLoaderLogger mirrorLog; private final LfsUpdateResult lfsUpdateResult; private final sonia.scm.repository.Repository repository; private final HttpConnectionFactory httpConnectionFactory; private EntryHandler(sonia.scm.repository.Repository repository, Repository gitRepository, - MirrorCommandRequest mirrorCommandRequest, - List mirrorLog, - LfsUpdateResult lfsUpdateResult) { + LfsLoaderLogger mirrorLog, + LfsUpdateResult lfsUpdateResult, + HttpConnectionFactory httpConnectionFactory) { this.lfsBlobStore = lfsBlobStoreFactory.getLfsBlobStore(repository); this.repository = repository; this.gitRepository = gitRepository; this.mirrorLog = mirrorLog; this.lfsUpdateResult = lfsUpdateResult; - this.httpConnectionFactory = mirrorHttpConnectionProvider.createHttpConnectionFactory(mirrorCommandRequest, mirrorLog); + this.httpConnectionFactory = httpConnectionFactory; } private void handleTreeEntry(TreeWalk treeWalk) { @@ -142,8 +146,7 @@ class LfsLoader { } } catch (Exception e) { LOG.warn("failed to load lfs file", e); - mirrorLog.add("Failed to load lfs file:"); - mirrorLog.add(e.getMessage()); + mirrorLog.failed(e); lfsUpdateResult.increaseFailureCount(); } } @@ -151,7 +154,7 @@ class LfsLoader { private Path loadLfsFile(LfsPointer lfsPointer) throws IOException { lfsUpdateResult.increaseOverallCount(); LOG.trace("trying to load lfs file '{}' for repository {}", lfsPointer.getOid(), repository); - mirrorLog.add(String.format("Loading lfs file with id '%s'", lfsPointer.getOid().name())); + mirrorLog.loading(lfsPointer.getOid().name()); Lfs lfs = new Lfs(gitRepository); lfs.getMediaFile(lfsPointer.getOid()); @@ -174,4 +177,11 @@ class LfsLoader { ); } } + + interface LfsLoaderLogger { + + void failed(Exception e); + + void loading(String name); + } } diff --git a/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/MirrorHttpConnectionProvider.java b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/MirrorHttpConnectionProvider.java index 88fbdaafad..bb4f4ea381 100644 --- a/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/MirrorHttpConnectionProvider.java +++ b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/MirrorHttpConnectionProvider.java @@ -24,7 +24,6 @@ package sonia.scm.repository.spi; -import org.eclipse.jgit.transport.UserAgent; import org.eclipse.jgit.transport.http.HttpConnectionFactory; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/PullHttpConnectionProvider.java b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/PullHttpConnectionProvider.java new file mode 100644 index 0000000000..ed5ef7070f --- /dev/null +++ b/scm-plugins/scm-git-plugin/src/main/java/sonia/scm/repository/spi/PullHttpConnectionProvider.java @@ -0,0 +1,55 @@ +/* + * MIT License + * + * Copyright (c) 2020-present Cloudogu GmbH and Contributors + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +package sonia.scm.repository.spi; + +import com.google.common.base.Strings; +import org.eclipse.jgit.transport.http.HttpConnectionFactory; +import sonia.scm.net.HttpConnectionOptions; +import sonia.scm.net.HttpURLConnectionFactory; +import sonia.scm.web.ScmHttpConnectionFactory; + +import javax.inject.Inject; +import java.nio.charset.StandardCharsets; +import java.util.Base64; + +class PullHttpConnectionProvider { + + private final HttpURLConnectionFactory httpURLConnectionFactory; + + @Inject + PullHttpConnectionProvider(HttpURLConnectionFactory httpURLConnectionFactory) { + this.httpURLConnectionFactory = httpURLConnectionFactory; + } + + HttpConnectionFactory createHttpConnectionFactory(PullCommandRequest request) { + HttpConnectionOptions options = new HttpConnectionOptions(); + if (!Strings.isNullOrEmpty(request.getUsername()) && !Strings.isNullOrEmpty(request.getPassword())) { + String encodedAuth = Base64.getEncoder().encodeToString((request.getUsername() + ":" + request.getPassword()).getBytes(StandardCharsets.UTF_8)); + String authHeaderValue = "Basic " + encodedAuth; + options.addRequestProperty("Authorization", authHeaderValue); + } + return new ScmHttpConnectionFactory(httpURLConnectionFactory, options); + } +} diff --git a/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitIncomingCommandTest.java b/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitIncomingCommandTest.java index a0121552ed..5d10230a97 100644 --- a/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitIncomingCommandTest.java +++ b/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitIncomingCommandTest.java @@ -40,6 +40,7 @@ import java.io.IOException; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; +import static org.mockito.Mockito.mock; //~--- JDK imports ------------------------------------------------------------ @@ -49,6 +50,9 @@ import static org.junit.Assert.assertNotNull; public class GitIncomingCommandTest extends AbstractRemoteCommandTestBase { + private final LfsLoader lfsLoader = mock(LfsLoader.class); + private final PullHttpConnectionProvider pullHttpConnectionProvider = mock(PullHttpConnectionProvider.class); + /** * Method description * @@ -99,7 +103,9 @@ public class GitIncomingCommandTest GitPullCommand pull = new GitPullCommand( handler, context, - postReceiveRepositoryHookEventFactory); + postReceiveRepositoryHookEventFactory, + lfsLoader, + pullHttpConnectionProvider); PullCommandRequest req = new PullCommandRequest(); req.setRemoteRepository(outgoingRepository); pull.pull(req); diff --git a/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitMirrorCommandTest.java b/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitMirrorCommandTest.java index cbc174f7bf..e479bed5d9 100644 --- a/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitMirrorCommandTest.java +++ b/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitMirrorCommandTest.java @@ -858,18 +858,17 @@ public class GitMirrorCommandTest extends AbstractGitCommandTestBase { // one revision is missing here ("fcd0ef1831e4002ac43ea539f4094334c79ea9ec"), because this is iterated twice, what is hard to test }).forEach(expectedRevision -> verify(lfsLoader) - .inspectTree(eq(ObjectId.fromString(expectedRevision)), any(), any(), any(), any(), eq(repository))); + .inspectTree(eq(ObjectId.fromString(expectedRevision)), any(), any(), any(), eq(repository), any(), any())); } @Test - public void shouldMarkMirrorAsFailedIfLfsFileFailes() { + public void shouldMarkMirrorAsFailedIfLfsFileFails() { doAnswer(invocation -> { - invocation.getArgument(4, MirrorCommandResult.LfsUpdateResult.class).increaseFailureCount(); + invocation.getArgument(3, MirrorCommandResult.LfsUpdateResult.class).increaseFailureCount(); return null; }) .when(lfsLoader) - .inspectTree(eq(ObjectId.fromString("a8495c0335a13e6e432df90b3727fa91943189a7")), any(), any(), any(), any(), eq(repository)); - + .inspectTree(eq(ObjectId.fromString("a8495c0335a13e6e432df90b3727fa91943189a7")), any(), any(), any(), eq(repository), any(), any()); MirrorCommandResult mirrorCommandResult = callMirrorCommand(); @@ -881,7 +880,7 @@ public class GitMirrorCommandTest extends AbstractGitCommandTestBase { callMirrorCommand(repositoryDirectory.getAbsolutePath(), c -> c.setIgnoreLfs(true)); verify(lfsLoader, never()) - .inspectTree(any(), any(), any(), any(), any(), any()); + .inspectTree(any(), any(), any(), any(), any(), any(), any()); } public static class DefaultBranchSelectorTest { diff --git a/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitModificationsCommandTest.java b/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitModificationsCommandTest.java index efe3888f2f..fd42581efc 100644 --- a/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitModificationsCommandTest.java +++ b/scm-plugins/scm-git-plugin/src/test/java/sonia/scm/repository/spi/GitModificationsCommandTest.java @@ -30,6 +30,7 @@ import org.eclipse.jgit.revwalk.RevCommit; import org.junit.Before; import org.junit.Test; import org.junit.runner.RunWith; +import org.mockito.Mock; import org.mockito.Mockito; import org.mockito.junit.MockitoJUnitRunner; import sonia.scm.repository.GitConfig; @@ -49,6 +50,11 @@ public class GitModificationsCommandTest extends AbstractRemoteCommandTestBase { private GitModificationsCommand incomingModificationsCommand; private GitModificationsCommand outgoingModificationsCommand; + @Mock + private LfsLoader lfsLoader; + @Mock + private PullHttpConnectionProvider pullHttpConnectionProvider; + @Before public void init() { incomingModificationsCommand = new GitModificationsCommand(Mockito.spy(new GitContext(incomingDirectory, incomingRepository, null, new GitConfig()))); @@ -171,7 +177,9 @@ public class GitModificationsCommandTest extends AbstractRemoteCommandTestBase { GitPullCommand pullCommand = new GitPullCommand( handler, context, - postReceiveRepositoryHookEventFactory); + postReceiveRepositoryHookEventFactory, + lfsLoader, + pullHttpConnectionProvider); PullCommandRequest pullRequest = new PullCommandRequest(); pullRequest.setRemoteRepository(incomingRepository); pullCommand.pull(pullRequest); diff --git a/scm-ui/ui-types/src/Notifications.ts b/scm-ui/ui-types/src/Notifications.ts index f23d72ca0e..767ff18785 100644 --- a/scm-ui/ui-types/src/Notifications.ts +++ b/scm-ui/ui-types/src/Notifications.ts @@ -29,6 +29,7 @@ export type Notification = HalRepresentation & { type: "INFO" | "SUCCESS" | "WARNING" | "ERROR"; link: string; message: string; + parameters?: Record; }; type EmbeddedNotifications = { diff --git a/scm-ui/ui-types/src/Repositories.ts b/scm-ui/ui-types/src/Repositories.ts index 4f3dd48c87..cb1377aa0f 100644 --- a/scm-ui/ui-types/src/Repositories.ts +++ b/scm-ui/ui-types/src/Repositories.ts @@ -60,6 +60,7 @@ export type RepositoryUrlImport = RepositoryCreation & { importUrl: string; username?: string; password?: string; + skipLfs?: boolean; }; export type ExportInfo = HalRepresentation & { diff --git a/scm-ui/ui-webapp/public/locales/de/repos.json b/scm-ui/ui-webapp/public/locales/de/repos.json index 80a9a5a691..9f6d5b80d1 100644 --- a/scm-ui/ui-webapp/public/locales/de/repos.json +++ b/scm-ui/ui-webapp/public/locales/de/repos.json @@ -30,7 +30,8 @@ "initializeRepository": "Erstellt einen ersten Branch und committet eine README.md.", "importUrlHelpText": "Importiert das gesamte Repository inkl. aller Branches und Tags über die Remote URL.", "usernameHelpText": "Benutzername könnte für den Import benötigt werden. Wird ignoriert, falls nicht gesetzt.", - "passwordHelpText": "Password könnte für den Import benötigt werden. Wird ignoriert, falls nicht gesetzt." + "passwordHelpText": "Passwort könnte für den Import benötigt werden. Wird ignoriert, falls nicht gesetzt.", + "skipLfsHelpText": "Ist diese Option aktiviert, werden beim Import keine (potentiell vorhandenen) LFS-Dateien in den SCM-Manager geladen. Diese Option ist nur für Git Repositories relevant." }, "repositoryRoot": { "errorTitle": "Fehler", @@ -115,7 +116,8 @@ "helpText": "Der Dump muss von einer SCM-Manager-Instanz mit Metadaten erstellt worden sein." } }, - "navigationWarning": "Es wird dringend empfohlen auf dieser Seite zu warten bis der Import abgeschlossen ist. Beim Verlassen der Seite könnte der Import abbrechen. Seite trotzdem verlassen?" + "navigationWarning": "Es wird dringend empfohlen auf dieser Seite zu warten bis der Import abgeschlossen ist. Beim Verlassen der Seite könnte der Import abbrechen. Seite trotzdem verlassen?", + "skipLfs": "Keine LFS Dateien laden (nur für Git Repositories)" }, "branches": { "overview": { diff --git a/scm-ui/ui-webapp/public/locales/en/repos.json b/scm-ui/ui-webapp/public/locales/en/repos.json index 7e005254ea..536f96813a 100644 --- a/scm-ui/ui-webapp/public/locales/en/repos.json +++ b/scm-ui/ui-webapp/public/locales/en/repos.json @@ -30,7 +30,8 @@ "initializeRepository": "Creates an initial branch and commits a basic README.md.", "importUrlHelpText": "Import the whole repository including all branches and tags via remote url", "usernameHelpText": "Username may be required to import the remote repository. Will be ignored if not provided.", - "passwordHelpText": "Password may be required to import the remote repository. Will be ignored if not provided." + "passwordHelpText": "Password may be required to import the remote repository. Will be ignored if not provided.", + "skipLfsHelpText": "Check this if (potentially available) LFS files shall not be loaded into SCM-Manager during the import. This option is relevant only for git repositories." }, "repositoryRoot": { "errorTitle": "Error", @@ -115,7 +116,8 @@ "helpText": "The dump containing the data must be generated by a SCM-Manager instance and include specific metadata." } }, - "navigationWarning": "It is strongly recommend that you stay on this page until the import is finished. Leaving this page could abort your import. Leave page anyway?" + "navigationWarning": "It is strongly recommend that you stay on this page until the import is finished. Leaving this page could abort your import. Leave page anyway?", + "skipLfs": "Do not load LFS files (git only)" }, "branches": { "overview": { diff --git a/scm-ui/ui-webapp/src/containers/Notifications.tsx b/scm-ui/ui-webapp/src/containers/Notifications.tsx index 4d746f071f..f7d6160593 100644 --- a/scm-ui/ui-webapp/src/containers/Notifications.tsx +++ b/scm-ui/ui-webapp/src/containers/Notifications.tsx @@ -71,8 +71,8 @@ const NotificationEntry: FC = ({ notification, removeToast }) => { } return ( - history.push(notification.link)} className="is-clickable"> - + history.push(notification.link) : undefined} className="is-clickable"> + @@ -175,9 +175,13 @@ const color = (notification: Notification) => { return c; }; -const NotificationMessage: FC<{ message: string }> = ({ message }) => { +const NotificationMessage: FC<{ message: string; parameters?: Record }> = ({ message, parameters }) => { const [t] = useTranslation("plugins"); - return t("notifications." + message, message); + if (parameters) { + return t("notifications." + message, message, parameters); + } else { + return t("notifications." + message, message); + } }; type SubscriptionProps = { @@ -185,6 +189,14 @@ type SubscriptionProps = { remove: (notification: Notification) => void; }; +const PotentialLink: FC<{ link?: string }> = ({ link, children }) => { + if (link) { + return {children}; + } else { + return <>{children}; + } +}; + const NotificationSubscription: FC = ({ notifications, remove }) => { const [t] = useTranslation("commons"); @@ -200,9 +212,9 @@ const NotificationSubscription: FC = ({ notifications, remove close={() => remove(notification)} >

- - - + + +

))} diff --git a/scm-ui/ui-webapp/src/repos/components/ImportFromUrlForm.tsx b/scm-ui/ui-webapp/src/repos/components/ImportFromUrlForm.tsx index 4ab75c82c7..202420394f 100644 --- a/scm-ui/ui-webapp/src/repos/components/ImportFromUrlForm.tsx +++ b/scm-ui/ui-webapp/src/repos/components/ImportFromUrlForm.tsx @@ -24,7 +24,7 @@ import React, { FC, useState } from "react"; import { useTranslation } from "react-i18next"; import { RepositoryUrlImport } from "@scm-manager/ui-types"; -import { InputField, validation } from "@scm-manager/ui-components"; +import { Checkbox, InputField, validation } from "@scm-manager/ui-components"; type Props = { repository: RepositoryUrlImport; @@ -87,6 +87,15 @@ const ImportFromUrlForm: FC = ({ repository, onChange, setValid, disabled disabled={disabled} /> +
+ onChange({ ...repository, skipLfs })} + checked={repository.skipLfs} + helpText={t("help.skipLfsHelpText")} + disabled={disabled} + /> +
); }; diff --git a/scm-webapp/src/main/java/sonia/scm/api/v2/resources/NotificationDto.java b/scm-webapp/src/main/java/sonia/scm/api/v2/resources/NotificationDto.java index ab0a6ee4ea..81fea689b8 100644 --- a/scm-webapp/src/main/java/sonia/scm/api/v2/resources/NotificationDto.java +++ b/scm-webapp/src/main/java/sonia/scm/api/v2/resources/NotificationDto.java @@ -31,6 +31,7 @@ import sonia.scm.notifications.StoredNotification; import sonia.scm.notifications.Type; import java.time.Instant; +import java.util.Map; @Data public class NotificationDto extends HalRepresentation { @@ -39,12 +40,14 @@ public class NotificationDto extends HalRepresentation { private Type type; private String link; private String message; + private Map parameters; public NotificationDto(StoredNotification notification, Links links) { super(links); this.type = notification.getType(); this.link = notification.getLink(); this.message = notification.getMessage(); + this.parameters = notification.getParameters(); this.createdAt = notification.getCreatedAt(); } } diff --git a/scm-webapp/src/main/java/sonia/scm/api/v2/resources/RepositoryImportResource.java b/scm-webapp/src/main/java/sonia/scm/api/v2/resources/RepositoryImportResource.java index b3b62e24b7..4d50a96df3 100644 --- a/scm-webapp/src/main/java/sonia/scm/api/v2/resources/RepositoryImportResource.java +++ b/scm-webapp/src/main/java/sonia/scm/api/v2/resources/RepositoryImportResource.java @@ -383,6 +383,7 @@ public class RepositoryImportResource { private String importUrl; private String username; private String password; + private boolean skipLfs; } @Getter @@ -400,6 +401,8 @@ public class RepositoryImportResource { String getUsername(); String getPassword(); + + boolean isSkipLfs(); } interface ImportRepositoryFromFileDto extends CreateRepositoryDto { diff --git a/scm-webapp/src/main/java/sonia/scm/importexport/FromUrlImporter.java b/scm-webapp/src/main/java/sonia/scm/importexport/FromUrlImporter.java index a93ca62c54..5851cfcecd 100644 --- a/scm-webapp/src/main/java/sonia/scm/importexport/FromUrlImporter.java +++ b/scm-webapp/src/main/java/sonia/scm/importexport/FromUrlImporter.java @@ -42,6 +42,7 @@ import sonia.scm.repository.RepositoryType; import sonia.scm.repository.api.Command; import sonia.scm.repository.api.ImportFailedException; import sonia.scm.repository.api.PullCommandBuilder; +import sonia.scm.repository.api.PullResponse; import sonia.scm.repository.api.RepositoryService; import sonia.scm.repository.api.RepositoryServiceFactory; @@ -63,13 +64,15 @@ public class FromUrlImporter { private final RepositoryServiceFactory serviceFactory; private final ScmEventBus eventBus; private final RepositoryImportLoggerFactory loggerFactory; + private final ImportNotificationHandler notificationHandler; @Inject - public FromUrlImporter(RepositoryManager manager, RepositoryServiceFactory serviceFactory, ScmEventBus eventBus, RepositoryImportLoggerFactory loggerFactory) { + public FromUrlImporter(RepositoryManager manager, RepositoryServiceFactory serviceFactory, ScmEventBus eventBus, RepositoryImportLoggerFactory loggerFactory, ImportNotificationHandler notificationHandler) { this.manager = manager; this.serviceFactory = serviceFactory; this.eventBus = eventBus; this.loggerFactory = loggerFactory; + this.notificationHandler = notificationHandler; } public Repository importFromUrl(RepositoryImportParameters parameters, Repository repository) { @@ -112,21 +115,35 @@ public class FromUrlImporter { .withUsername(parameters.getUsername()) .withPassword(parameters.getPassword()); } + pullCommand.doFetchLfs(!parameters.isSkipLfs()); logger.step("pulling repository from " + parameters.getImportUrl()); - pullCommand.pull(parameters.getImportUrl()); + PullResponse pullResponse = pullCommand.pull(parameters.getImportUrl()); logger.finished(); + handle(pullResponse, repository); } catch (IOException e) { throw new InternalRepositoryException(repository, "Failed to import from remote url: " + e.getMessage(), e); + } catch (ImportFailedException e) { + notificationHandler.handleFailedImport(); + throw e; } }; } + private void handle(PullResponse pullResponse, Repository repository) { + if (pullResponse.getLfsCount().getFailureCount() == 0) { + notificationHandler.handleSuccessfulImport(repository, pullResponse.getLfsCount()); + } else { + notificationHandler.handleSuccessfulImportWithLfsFailures(repository, pullResponse.getLfsCount()); + } + } + @Getter @Setter public static class RepositoryImportParameters { private String importUrl; private String username; private String password; + private boolean skipLfs; } } diff --git a/scm-webapp/src/main/java/sonia/scm/importexport/ImportNotificationHandler.java b/scm-webapp/src/main/java/sonia/scm/importexport/ImportNotificationHandler.java new file mode 100644 index 0000000000..18274c24a3 --- /dev/null +++ b/scm-webapp/src/main/java/sonia/scm/importexport/ImportNotificationHandler.java @@ -0,0 +1,88 @@ +/* + * MIT License + * + * Copyright (c) 2020-present Cloudogu GmbH and Contributors + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +package sonia.scm.importexport; + +import sonia.scm.notifications.Notification; +import sonia.scm.notifications.NotificationSender; +import sonia.scm.notifications.Type; +import sonia.scm.repository.Repository; +import sonia.scm.repository.api.PullResponse; + +import javax.inject.Inject; +import java.util.Map; + +class ImportNotificationHandler { + + private final NotificationSender notificationSender; + + @Inject + ImportNotificationHandler(NotificationSender notificationSender) { + this.notificationSender = notificationSender; + } + + void handleSuccessfulImport(Repository repository) { + handleSuccessfulImport(repository, new PullResponse.LfsCount(0, 0)); + } + + void handleSuccessfulImport(Repository repository, PullResponse.LfsCount lfsCount) { + notificationSender.send(getImportSuccessfulNotification(repository, lfsCount)); + } + + void handleSuccessfulImportWithLfsFailures(Repository repository, PullResponse.LfsCount lfsCount) { + notificationSender.send(getImportLfsFailedNotification(repository, lfsCount)); + } + + void handleFailedImport() { + notificationSender.send(getImportFailedNotification()); + } + + private Notification getImportSuccessfulNotification(Repository repository, PullResponse.LfsCount lfsCount) { + if (lfsCount.getSuccessCount() > 0 || lfsCount.getFailureCount() > 0) { + return new Notification(Type.SUCCESS, createLink(repository), "importWithLfsFinished", createParameters(lfsCount)); + } else { + return new Notification(Type.SUCCESS, createLink(repository), "importFinished"); + } + } + + private Notification getImportLfsFailedNotification(Repository repository, PullResponse.LfsCount lfsCount) { + return new Notification(Type.ERROR, createLink(repository), "importLfsFailed", createParameters(lfsCount)); + } + + private static Map createParameters(PullResponse.LfsCount lfsCount) { + return Map.of( + "successCount", Integer.toString(lfsCount.getSuccessCount()), + "failureCount", Integer.toString(lfsCount.getFailureCount()), + "overallCount", Integer.toString(lfsCount.getSuccessCount() + lfsCount.getFailureCount()) + ); + } + + private Notification getImportFailedNotification() { + return new Notification(Type.ERROR, null, "importFailed"); + } + + private static String createLink(Repository repository) { + return "/repo/" + repository.getNamespaceAndName(); + } +} diff --git a/scm-webapp/src/main/java/sonia/scm/notifications/StoredNotification.java b/scm-webapp/src/main/java/sonia/scm/notifications/StoredNotification.java index cb38f03ea5..b9252edd55 100644 --- a/scm-webapp/src/main/java/sonia/scm/notifications/StoredNotification.java +++ b/scm-webapp/src/main/java/sonia/scm/notifications/StoredNotification.java @@ -33,6 +33,7 @@ import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessorType; import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter; import java.time.Instant; +import java.util.Map; @Data @NoArgsConstructor @@ -44,6 +45,7 @@ public class StoredNotification { Type type; String link; String message; + Map parameters; @XmlJavaTypeAdapter(XmlInstantAdapter.class) Instant createdAt; @@ -53,5 +55,6 @@ public class StoredNotification { this.type = notification.getType(); this.link = notification.getLink(); this.message = notification.getMessage(); + this.parameters = notification.getParameters(); } } diff --git a/scm-webapp/src/main/resources/locales/de/plugins.json b/scm-webapp/src/main/resources/locales/de/plugins.json index f43be776bd..6cf3eb9b7c 100644 --- a/scm-webapp/src/main/resources/locales/de/plugins.json +++ b/scm-webapp/src/main/resources/locales/de/plugins.json @@ -480,6 +480,10 @@ "notifications": { "exportFinished": "Der Repository Export wurde abgeschlossen.", "exportFailed": "Der Repository Export ist fehlgeschlagen. Versuchen Sie es erneut oder wenden Sie sich an einen Administrator.", + "importFinished": "Der Repository Import wurde abgeschlossen.", + "importWithLfsFinished": "Der Repository Import wurde abgeschlossen. Dabei wurden {{successCount}} LFS Dateien importiert.", + "importLfsFailed": "Der Repository Import wurde abgeschlossen. Jedoch konnten {{failureCount}} von {{overallCount}} LFS Dateien nicht geladen werden.", + "importFailed": "Der Repository Import ist fehlgeschlagen. Versuchen Sie es erneut oder wenden Sie sich an einen Administrator.", "healthCheckFailed": "Der Repository Health Check ist fehlgeschlagen.", "healthCheckSuccess": "Der Repository Health Check war erfolgreich." }, diff --git a/scm-webapp/src/main/resources/locales/en/plugins.json b/scm-webapp/src/main/resources/locales/en/plugins.json index 28a962d99e..3511895ccd 100644 --- a/scm-webapp/src/main/resources/locales/en/plugins.json +++ b/scm-webapp/src/main/resources/locales/en/plugins.json @@ -424,6 +424,10 @@ "notifications": { "exportFinished": "The repository export has been finished.", "exportFailed": "The repository export has failed. Try it again or contact your administrator.", + "importFinished": "The repository import has been finished.", + "importWithLfsFinished": "The repository import has been finished, including {{successCount}} LFS files.", + "importLfsFailed": "The repository import has been finished. There were {{failureCount}} errors during the import of {{overallCount}} LFS files.", + "importFailed": "The repository import has failed. Try it again or contact your administrator.", "healthCheckFailed": "The repository health check has failed.", "healthCheckSuccess": "The repository health check was successful." }, diff --git a/scm-webapp/src/test/java/sonia/scm/api/v2/resources/NotificationResourceTest.java b/scm-webapp/src/test/java/sonia/scm/api/v2/resources/NotificationResourceTest.java index ea8a7b2946..fdd6e949c2 100644 --- a/scm-webapp/src/test/java/sonia/scm/api/v2/resources/NotificationResourceTest.java +++ b/scm-webapp/src/test/java/sonia/scm/api/v2/resources/NotificationResourceTest.java @@ -33,7 +33,6 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; import org.mockito.Mock; import org.mockito.junit.jupiter.MockitoExtension; -import sonia.scm.notifications.Notification; import sonia.scm.notifications.NotificationStore; import sonia.scm.notifications.StoredNotification; import sonia.scm.notifications.Type; @@ -50,6 +49,7 @@ import java.util.List; import java.util.UUID; import java.util.stream.Collectors; +import static java.util.Collections.emptyMap; import static org.assertj.core.api.Assertions.assertThat; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; @@ -162,7 +162,7 @@ class NotificationResourceTest { } private StoredNotification notification(String m) { - return new StoredNotification(UUID.randomUUID().toString(), Type.INFO, "/notify", m, Instant.now()); + return new StoredNotification(UUID.randomUUID().toString(), Type.INFO, "/notify", m, emptyMap(), Instant.now()); } @Path("/api/v2") diff --git a/scm-webapp/src/test/java/sonia/scm/importexport/FromUrlImporterTest.java b/scm-webapp/src/test/java/sonia/scm/importexport/FromUrlImporterTest.java index 1a3d719ae6..c132aef023 100644 --- a/scm-webapp/src/test/java/sonia/scm/importexport/FromUrlImporterTest.java +++ b/scm-webapp/src/test/java/sonia/scm/importexport/FromUrlImporterTest.java @@ -28,8 +28,10 @@ import org.apache.shiro.subject.Subject; import org.apache.shiro.util.ThreadContext; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Answers; import org.mockito.InjectMocks; import org.mockito.Mock; import org.mockito.junit.jupiter.MockitoExtension; @@ -42,6 +44,7 @@ import sonia.scm.repository.RepositoryTestData; import sonia.scm.repository.RepositoryType; import sonia.scm.repository.api.ImportFailedException; import sonia.scm.repository.api.PullCommandBuilder; +import sonia.scm.repository.api.PullResponse; import sonia.scm.repository.api.RepositoryService; import sonia.scm.repository.api.RepositoryServiceFactory; @@ -54,7 +57,7 @@ import static org.junit.Assert.assertThrows; import static org.mockito.ArgumentMatchers.any; import static org.mockito.ArgumentMatchers.anyString; import static org.mockito.ArgumentMatchers.argThat; -import static org.mockito.Mockito.RETURNS_SELF; +import static org.mockito.ArgumentMatchers.eq; import static org.mockito.Mockito.doThrow; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.verify; @@ -78,11 +81,18 @@ class FromUrlImporterTest { private RepositoryImportLogger logger; @Mock private Subject subject; + @Mock(answer = Answers.RETURNS_SELF) + private PullCommandBuilder pullCommandBuilder; + @Mock + private ImportNotificationHandler notificationHandler; @InjectMocks private FromUrlImporter importer; private final Repository repository = RepositoryTestData.createHeartOfGold("git"); + private Repository createdRepository; + + private PullResponse mockedResponse = new PullResponse(); @BeforeEach void setUpMocks() { @@ -91,12 +101,13 @@ class FromUrlImporterTest { when(manager.create(any(), any())).thenAnswer( invocation -> { Repository repository = invocation.getArgument(0, Repository.class); - Repository createdRepository = repository.clone(); + createdRepository = repository.clone(); createdRepository.setNamespace("created"); invocation.getArgument(1, Consumer.class).accept(createdRepository); return createdRepository; } ); + when(service.getPullCommand()).thenReturn(pullCommandBuilder); } @BeforeEach @@ -119,50 +130,92 @@ class FromUrlImporterTest { ThreadContext.unbindSubject(); } - @Test - void shouldPullChangesFromRemoteUrl() throws IOException { - PullCommandBuilder pullCommandBuilder = mock(PullCommandBuilder.class, RETURNS_SELF); - when(service.getPullCommand()).thenReturn(pullCommandBuilder); + @Nested + class ForSuccessfulImports { - FromUrlImporter.RepositoryImportParameters parameters = new FromUrlImporter.RepositoryImportParameters(); - parameters.setImportUrl("https://scm-manager.org/scm/repo/scmadmin/scm-manager.git"); + @BeforeEach + void mockImportResult() throws IOException { + when(pullCommandBuilder.pull(anyString())).thenAnswer(invocation -> mockedResponse); + } - Repository createdRepository = importer.importFromUrl(parameters, repository); + @Test + void shouldPullChangesFromRemoteUrl() throws IOException { + FromUrlImporter.RepositoryImportParameters parameters = new FromUrlImporter.RepositoryImportParameters(); + parameters.setImportUrl("https://scm-manager.org/scm/repo/scmadmin/scm-manager.git"); - assertThat(createdRepository.getNamespace()).isEqualTo("created"); - verify(pullCommandBuilder).pull("https://scm-manager.org/scm/repo/scmadmin/scm-manager.git"); - verify(logger).finished(); - verify(eventBus).post(argThat( - event -> { - assertThat(event).isInstanceOf(RepositoryImportEvent.class); - RepositoryImportEvent repositoryImportEvent = (RepositoryImportEvent) event; - assertThat(repositoryImportEvent.getItem().getNamespace()).isEqualTo("created"); - assertThat(repositoryImportEvent.isFailed()).isFalse(); - return true; - } - )); - } + Repository createdRepository = importer.importFromUrl(parameters, repository); - @Test - void shouldPullChangesFromRemoteUrlWithCredentials() { - PullCommandBuilder pullCommandBuilder = mock(PullCommandBuilder.class, RETURNS_SELF); - when(service.getPullCommand()).thenReturn(pullCommandBuilder); + assertThat(createdRepository.getNamespace()).isEqualTo("created"); + verify(pullCommandBuilder).pull("https://scm-manager.org/scm/repo/scmadmin/scm-manager.git"); + verify(logger).finished(); + verify(eventBus).post(argThat( + event -> { + assertThat(event).isInstanceOf(RepositoryImportEvent.class); + RepositoryImportEvent repositoryImportEvent = (RepositoryImportEvent) event; + assertThat(repositoryImportEvent.getItem().getNamespace()).isEqualTo("created"); + assertThat(repositoryImportEvent.isFailed()).isFalse(); + return true; + } + )); + verify(notificationHandler) + .handleSuccessfulImport( + eq(createdRepository), + argThat(argument -> argument.getSuccessCount() == 0)); + } - FromUrlImporter.RepositoryImportParameters parameters = new FromUrlImporter.RepositoryImportParameters(); - parameters.setImportUrl("https://scm-manager.org/scm/repo/scmadmin/scm-manager.git"); - parameters.setUsername("trillian"); - parameters.setPassword("secret"); + @Test + void shouldPullChangesFromRemoteUrlWithCredentials() { + FromUrlImporter.RepositoryImportParameters parameters = new FromUrlImporter.RepositoryImportParameters(); + parameters.setImportUrl("https://scm-manager.org/scm/repo/scmadmin/scm-manager.git"); + parameters.setUsername("trillian"); + parameters.setPassword("secret"); - importer.importFromUrl(parameters, repository); + importer.importFromUrl(parameters, repository); - verify(pullCommandBuilder).withUsername("trillian"); - verify(pullCommandBuilder).withPassword("secret"); + verify(pullCommandBuilder).withUsername("trillian"); + verify(pullCommandBuilder).withPassword("secret"); + } + + @Test + void shouldPullChangesWithLfsIfNotDisabled() { + FromUrlImporter.RepositoryImportParameters parameters = new FromUrlImporter.RepositoryImportParameters(); + parameters.setImportUrl("https://scm-manager.org/scm/repo/scmadmin/scm-manager.git"); + parameters.setSkipLfs(false); + + importer.importFromUrl(parameters, repository); + + verify(pullCommandBuilder).doFetchLfs(true); + } + + @Test + void shouldPullChangesWithoutLfsIfSelected() { + FromUrlImporter.RepositoryImportParameters parameters = new FromUrlImporter.RepositoryImportParameters(); + parameters.setImportUrl("https://scm-manager.org/scm/repo/scmadmin/scm-manager.git"); + parameters.setSkipLfs(true); + + importer.importFromUrl(parameters, repository); + + verify(pullCommandBuilder).doFetchLfs(false); + } + + @Test + void shouldHandleFailedLfsFiles() { + FromUrlImporter.RepositoryImportParameters parameters = new FromUrlImporter.RepositoryImportParameters(); + parameters.setImportUrl("https://scm-manager.org/scm/repo/scmadmin/scm-manager.git"); + parameters.setSkipLfs(false); + mockedResponse = new PullResponse(42, new PullResponse.LfsCount(0, 1)); + + importer.importFromUrl(parameters, repository); + + verify(notificationHandler). + handleSuccessfulImportWithLfsFailures( + eq(createdRepository), + argThat(argument -> argument.getFailureCount() == 1 && argument.getSuccessCount() == 0)); + } } @Test void shouldThrowImportFailedEvent() throws IOException { - PullCommandBuilder pullCommandBuilder = mock(PullCommandBuilder.class, RETURNS_SELF); - when(service.getPullCommand()).thenReturn(pullCommandBuilder); doThrow(TestException.class).when(pullCommandBuilder).pull(anyString()); when(logger.started()).thenReturn(true); diff --git a/scm-webapp/src/test/java/sonia/scm/importexport/ImportNotificationHandlerTest.java b/scm-webapp/src/test/java/sonia/scm/importexport/ImportNotificationHandlerTest.java new file mode 100644 index 0000000000..9c4fd47a12 --- /dev/null +++ b/scm-webapp/src/test/java/sonia/scm/importexport/ImportNotificationHandlerTest.java @@ -0,0 +1,110 @@ +/* + * MIT License + * + * Copyright (c) 2020-present Cloudogu GmbH and Contributors + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +package sonia.scm.importexport; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import sonia.scm.notifications.NotificationSender; +import sonia.scm.notifications.Type; +import sonia.scm.repository.Repository; +import sonia.scm.repository.RepositoryTestData; +import sonia.scm.repository.api.PullResponse; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.argThat; +import static org.mockito.Mockito.verify; + +@ExtendWith(MockitoExtension.class) +class ImportNotificationHandlerTest { + + @Mock + private NotificationSender notificationSender; + @InjectMocks + private ImportNotificationHandler handler; + + private final Repository repository = RepositoryTestData.createHeartOfGold(); + + @Test + void shouldCreateSuccessNotification() { + handler.handleSuccessfulImport(repository); + + verify(notificationSender).send(argThat( + notification -> { + assertThat(notification.getType()).isEqualTo(Type.SUCCESS); + assertThat(notification.getMessage()).isEqualTo("importFinished"); + assertThat(notification.getParameters()).isNull(); + return true; + } + )); + } + + @Test + void shouldCreateSuccessNotificationWithLfs() { + handler.handleSuccessfulImport(repository, new PullResponse.LfsCount(42, 0)); + + verify(notificationSender).send(argThat( + notification -> { + assertThat(notification.getType()).isEqualTo(Type.SUCCESS); + assertThat(notification.getMessage()).isEqualTo("importWithLfsFinished"); + assertThat(notification.getParameters()).containsEntry("successCount", "42"); + return true; + } + )); + } + + @Test + void shouldCreateFailureNotification() { + handler.handleFailedImport(); + + verify(notificationSender).send(argThat( + notification -> { + assertThat(notification.getType()).isEqualTo(Type.ERROR); + assertThat(notification.getMessage()).isEqualTo("importFailed"); + assertThat(notification.getParameters()).isNull(); + return true; + } + )); + } + + @Test + void shouldCreateLfsFailureNotification() { + handler.handleSuccessfulImportWithLfsFailures(repository, new PullResponse.LfsCount(42, 7)); + + verify(notificationSender).send(argThat( + notification -> { + assertThat(notification.getType()).isEqualTo(Type.ERROR); + assertThat(notification.getMessage()).isEqualTo("importLfsFailed"); + assertThat(notification.getParameters()) + .containsEntry("successCount", "42") + .containsEntry("failureCount", "7") + .containsEntry("overallCount", "49"); + return true; + } + )); + } +}